Debuggers & Domains

Words About Computing - Aaron Dandy

Making Azure Functions Work for Large Responses

For various reasons, some of us Azure Functions users out there need to return larger HTTP responses. In my case, I needed to create an intelligent proxy that returns some large and small files based on SRSBSNS rules. If you have tried this like me, you may have also noticed some odd qualities in your responses. For smaller content, you may have experienced long delays before the first bytes reached a client, and then slow transfer speeds after that. For larger content you may have noticed your seemingly successful HTTP 200 responses were empty or contained some UUID value instead of your actual content. If you were lucky, unlike me, you would have also noticed the OutOfMemoryException before another business pointed the problem out to you. Oh shucks, lets dig into this!

Reproduction

Thankfully this issue can be reproduced locally, lets try that out:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

// 💩🚽💩

[FunctionName("FancyProxy")]
public static IActionResult Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)] HttpRequest req)
{
    // TODO: SRSBSNS
    return new BustedProxyResult("https://download.microsoft.com/download/1/9/8/1986c4a9-480b-4a46-8088-2778e0abcc8a/SSMS-Setup-ENU.exe");
}

private class BustedProxyResult : IActionResult
{
    private static readonly HttpClient Client = new HttpClient();

    public BustedProxyResult(string sourceUrl)
    {
        this.sourceUrl = sourceUrl ?? throw new ArgumentNullException(nameof(sourceUrl));
    }

    private readonly string sourceUrl;

    public async Task ExecuteResultAsync(ActionContext context)
    {
        using var sourceResponse = await Client.GetAsync(sourceUrl, HttpCompletionOption.ResponseHeadersRead, context.HttpContext.RequestAborted);
        sourceResponse.EnsureSuccessStatusCode();
        context.HttpContext.Response.StatusCode = (int)sourceResponse.StatusCode;
        context.HttpContext.Response.ContentType = sourceResponse.Content.Headers.ContentType?.ToString();
        context.HttpContext.Response.ContentLength = sourceResponse.Content.Headers.ContentLength;

        using var sourceStream = await sourceResponse.Content.ReadAsStreamAsync();
        await sourceStream.CopyToAsync(context.HttpContext.Response.Body, context.HttpContext.RequestAborted);
    }
}

After invoking this function locally, you can watch your browser sit there for a few minutes as it waits for a response. Eventually you may get a 5xx response and see this in your func.exe console window:

An unhandled host error has occurred. System.Private.CoreLib: Exception of type 'System.OutOfMemoryException' was thrown.

Worse yet because we set the HTTP response status code to a success status code before proxying the data, when run in Azure you may get an HTTP 200 response that contains some UUID value or that will be blank. Hopefully you get a 502 response instead. This can be a pretty perplexing failure when encountered as it sure looks like a success sometimes.

[Information] Executed 'FancyProxy' (Succeeded, Id=<some-other-uuid>)

And just in case you were wondering, this also impacts proxies defined in a proxies.json file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

{
  "$schema": "http://json.schemastore.org/proxies",
  "proxies": {
    "TestProxy": {
      "matchCondition": {
        "route": "/azfn-proxy",
        "methods": [ "GET" ]
      },
      "backendUri": "http://download.microsoft.com/download/4/3/B/43B61315-B2CE-4F5B-9E32-34CCA07B2F0E/NDP452-KB2901951-x86-x64-DevPack.exe"
    }
  }
}

Solution

It seems like we are waiting for some buffer to fill up before it returns our first bytes. That OutOfMemoryException we get also indicates that we have reached our memory limit, probably due to buffering. Looking at what I think is the source for this behavior, a workaround looks possible:

If we can provide a courtesy flush before we release our content into the response stream, we can deactivate the buffering that is causing our bowl to overflow.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

// 🌊🏄🚽

using System;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;

// ...

[FunctionName("FancyProxy")]
public static IActionResult Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)] HttpRequest req)
{
    // TODO: SRSBSNS
    return new ProxyWithCourtesyFlushResult("https://download.microsoft.com/download/1/9/8/1986c4a9-480b-4a46-8088-2778e0abcc8a/SSMS-Setup-ENU.exe");
}

private class ProxyWithCourtesyFlushResult : IActionResult
{
    private static readonly HttpClient Client = new HttpClient();

    public ProxyWithCourtesyFlushResult(string sourceUrl)
    {
        this.sourceUrl = sourceUrl ?? throw new ArgumentNullException(nameof(sourceUrl));
    }

    private readonly string sourceUrl;

    public async Task ExecuteResultAsync(ActionContext context)
    {
        using var sourceResponse = await Client.GetAsync(sourceUrl, HttpCompletionOption.ResponseHeadersRead, context.HttpContext.RequestAborted);
        sourceResponse.EnsureSuccessStatusCode();
        context.HttpContext.Response.StatusCode = (int)sourceResponse.StatusCode;
        context.HttpContext.Response.ContentType = sourceResponse.Content.Headers.ContentType?.ToString();
        context.HttpContext.Response.ContentLength = sourceResponse.Content.Headers.ContentLength;

        using var sourceStream = await sourceResponse.Content.ReadAsStreamAsync();
        // NOTE: This courtesy flush deactivates buffering
        await context.HttpContext.Response.Body.FlushAsync(context.HttpContext.RequestAborted);
        await sourceStream.CopyToAsync(context.HttpContext.Response.Body, context.HttpContext.RequestAborted);
    }
}

Conclusion

The most important result of this workaround is that we can now actually return larger responses from Azure Functions without experiencing any confusing failures. If you aren’t running into failures, there is also a cost consideration here. After some shoddy testing and quick calculations this workaround seems to also reduce the total function execution time by nearly 75% in some cases. This can be the difference between a free month of execution and a whopping $10 USD. Well, I’m sure that cost adds up for somebody. More seriously, if you attempted to solve this problem using more memory or a dedicated app service plan it could cost a great deal more and yet still possibly fail. Bypassing the buffering is vital for these kinds of responses and could save you some money 💹. I don’t understand what problem this buffering solves, but from my point of view it is just a source of trouble.