Ever since codebender started growing, one of the biggest challenges we faced was pushing updates to our compiler and being able to quickly iterate on it, fixing bugs and improving its performance. Since we're kind of obsessed with making all of codebender's users happy, we need to make sure that all changes applied to the compiler won't break existing sketches. After all, what's a web Arduino IDE if you can't compile your code in it? As of today, about 70000 users have registered to codebender and it hosts more than 200000 projects. How can you make sure all of them work fine without having the whole testing process last forever?
The initial idea
At first, all we had was a PHP script that fetched codebender's projects directly from MongoDB where they are stored and sent each project's code to the compiler using cURL. We would then gather the results and create a report regarding which projects compiled successfully before the update, and which failed to compile after the update. This seemed promising and was pretty fast as long as the number of projects was small. New users kept registering to codebender and the number of projects soon could not be managed by our testing script.
Guzzle to the rescue
We obviously had to start compiling several projects in parallel and plain cURL was far from efficient in such a task. Luckily Guzzle could do that for us in a pretty neat way. Guzzle is an easy-to-use PHP HTTP client with very useful features such as persistent connections, parallel requests, streams and efficient handling of HTTP requests and responses. You can install it to any PHP project using Composer Installer. Let's see a simple example.
<?php
use GuzzleHttp\Client;
use GuzzleHttp\Psr7\Request;
/*
* Create a Client object. By default all requests are sent to the `base_uri`
* You can use the same Client instance multiple times,
* no need to re-initialize it
*/
$client = new Client(['base_uri' => 'http://codebender/compiler/endpoint']);
/*
* First parameter is the HTTP method used
* We don't use any specific URI, so the second parameter is left empty.
* The third parameter is an assoc array containing all the request attributes (headers, body, etc)
*/
$request = new Request('POST', '', ['body' => 'The body of the request as a string']);
$response = $client->send($request);
/*
* Handle the response here
*/
$statusCode = $response->getStatusCode();
$responseBody = $response->getBody->getContents();
Easy, right? Notice that although Guzzle uses cURL (default HTTP client), we didn't have to provide any configuration other than the base url of the compiler. Just plug and play. 😄
Performing concurrent requests
Moving on to performing multiple requests to the compiler server, we obviously can't just perform all requests at once. We'll have to create batches containing multiple request objects and send them to the compiler using Guzzle's Pool.
<?php
use GuzzleHttp\Client;
use GuzzleHttp\Pool;
use GuzzleHttp\Psr7\Request;
// Initialize the Client
$client = new Client(['base_uri' => 'http://codebender/compiler/endpoint']);
// The `projects` array contains the code of 60 Arduino projects
$projects = ['project1' => ['code' => '...'], 'project2' => ['code' => '...'], ...];
$requests = [];
// Gather the requests for all provided projects
foreach ($projects as $project) {
// create the body of the request
// get the project files & their contents as a JSON string
$body = json_encode($project['code']);
$requests[] = new Request('POST', '', ['body' => $body]);
}
// Perform the actual requests
$responses = Pool::batch($this->client, $requests, ['concurrency' => 60]);
foreach ($responses as $response) {
// Check the response data
}
In this case, all 60 requests are performed asynchronously and the Client waits until each of them is completed or has reached its timeout (if you have set any). As long as you have plenty of resources, you can increase the concurrency to whatever value best suits your needs.
Guzzle did help us quite a lot in the quality assurance process of codebender's compiler, but it's not all about parallel requests. It's a state-of-the-art PHP library that can handle almost anything you might need to do over HTTP. It's used by PHP frameworks such as Symfony2 and Amazon's PHP SDK is based on Guzzle. If you're interested in learning more about it, take a look at the latest documentation and start coding!