Laravel working with Queues

What are queues in web development?

• Queues allow you to defer/postpone the processing of a time consuming task, such as sending an e-mail, until a later time, thus drastically speeding up the web requests to your application, which returns a response and serves the client significantly faster than it would be if the task ran synchronously.
• The Laravel Queue component provides a unified API across a variety of different queue services. There are different queue services. Some of them are hosted elsewhere, but some of them can be self-hosted, such as Beanstalkd, which I'll cover here.

How do they work behind the scenes?

• Once you push a job onto the queue (from somewhere within your code, e.g. send an email after user has registered), all that the client has to wait for is for job to be pushed onto the queue service that is listening for jobs, and doesn't have to wait for the queue service to finish the job itself.
• The most trivial example would be a McDonalds employee working at the cashier - the employee asks you for your order, but it doesn't go and make french-fries and hamburgers for you. It delegates the task to someone in the background (to people which are making hamburgers and french-fries), but the cashier employee returns immediatelly to you, and in the end delivers you the french-fries once they're ready.

Queue drivers in Laravel

There are several drivers for queues in Laravel. There are some differences between them, and I'll try to explain them as best as I could, usually the biggest difference being how the jobs are stored by the queueing service.

database driver

• In order to use the database queue driver, you will need a database table to hold the jobs. To generate a migration to create this table, run the queue:table Artisan command.
• For every job that you push onto the queue, it will get stored into the database. What this means is it uses I/O operations on the filesystem, which might not be the fastest thing in the world. It depends how fast you want your jobs to run.
  • pros: uses disk storage which is very cheap, thus meaning you could have a really huge number of jobs in the queue
  • cons: a lot slower than e.g. some driver that reads/writes to memory, as it has to read/write onto the filesystem

beanstalkd driver

• In order to use this driver, you have to pull in the pda/pheanstalk ~3.0 package and have beanstalkd installed somewhere. You can install it on the same server as where you Laravel app is, or you can have a dedicated server only for listening and running queued jobs. Either one works.
  • pros: is an in-memory queue service, which means it's blazing fast. It's also free and you can host it yourself.
  • cons: well, eventually you run out of memory once there are too many jobs on the queue.
    • Hint: you may run Supervisor that would re-start the process once it's down, or you can use the -b option, and beanstalkd will write all jobs to a binlog. If something odd happens, you can restart beanstalkd with the same option and it will recover the contents of the log.
• Pretty much the same pros and cons are for redis driver as well.

iron driver

• If you do not wish to bother installing and hosting queue services such as beanstalkd, then you might pay and use external ones, such as iron.io. It basically does the same thing, except you don't have to worry about crashes and exceeding memory limit etc.
  • pros: it's very reliable solution. You don't have to worry about setting up anything.
  • cons: it costs money, for one. The synced job has to be pushed onto a cloud, and retrieved back from the cloud. Sometimes it might take time, so not the fastest thing in the world either.

sync driver

• It is basically intended for development purposes usually. It just runs the queued jobs synchronously, just as if you haven't pushed them onto a queue service. Used so you can develop in a familiar API and work with queues, and then ideally when you want it in production you should just specify in the config which driver you want to use and everything should automatically work.

null driver

• It just doesn't run the jobs you push to the queue using the API.

Listening for jobs

Once everything set-up, you'd have to run queue:listen artisan command in order to start listening for jobs. If you don't, the jobs would be pushed onto the queue but never actually executed.
You might as well try to run queue:work with --daemon option for forcing the queue worker to continue processing jobs without ever re-booting the framework. This results in a significant reduction of CPU usage when compared to the queue:listen command, but at the added complexity of needing to drain the queues of currently executing jobs during your deployments.
Daemon queue workers do not restart the framework before processing each job. Therefore, you should be careful to free any heavy resources before your job finishes. For example, if you are doing image manipulation with the GD library, you should free the memory with imagedestroy when you are done.
Similarly, your database connection may disconnect when being used by long-running daemon. You may use the DB::reconnect method to ensure you have a fresh connection.
Hope it was helpful.