I’m looking at starting a service that involves hosting a lot of LLM models, which are often going to be 16GB+ (compressed). I did a bit of searching for cloud storage providers with cheap egress, and the cheapest I could find is $0.01 per GB, which would still be $0.16+ per download.

How do sites like Huggingface or CivitAI do it? Lots of VC funding?

  • Oliver Lowe@lemmy.sdf.orgEnglish
    4·
    1 year ago

    If the files are not going to be changing much, then what is typically done is to use a CDN service (e.g. Cloudflare, Akamai, Fastly). The idea is you have an “origin” which could be any old server which serves your files over HTTP (even a VPS running nginx). The CDN is configured to proxy requests to the origin, building up a cache of the files it serves. The CDN can serve files from cache on their own (very large) infrastructure. See also What is a CDN?

    • Oliver Lowe@lemmy.sdf.orgEnglish
      5·
      1 year ago

      So I got curious and wondered how HuggingFace hosts their files. It’s AWS CloudFront:

      $ curl -LI https://huggingface.co/facebook/musicgen-large/resolve/main/state_dict.bin

HTTP/2 302 
content-type: text/plain; charset=utf-8
content-length: 1198
location: https://cdn-lfs.huggingface.co/repos/81/...
date: Fri, 23 Jun 2023 07:09:39 GMT
x-powered-by: huggingface-moon
x-request-id: Root=1-64954533-1eb79eed4ea500203f6435cb
access-control-allow-origin: https://huggingface.co
vary: Origin, Accept
access-control-expose-headers: X-Repo-Commit,X-Request-Id,X-Error-Code,X-Error-Message,ETag,Link,Accept-Ranges,Content-Range
x-repo-commit: c19300a6b2b62d29b345ae9eb7b163278e65238a
accept-ranges: bytes
x-linked-size: 6514108999
x-linked-etag: "1f0cf17b5e65c5dd8ba71767371c377f174e4ce1db44bc4d6657825769f26ffd"
x-cache: Miss from cloudfront
via: 1.1 65c7d0c3355767ac8658c2122c8280b6.cloudfront.net (CloudFront)
x-amz-cf-pop: SYD1-C1
x-amz-cf-id: XAU8X4yneUeudylCi_9MeAYZmISCr8OHiBcgjAGcBQT-edrBF6wGCA==

HTTP/2 200 
content-type: application/octet-stream
content-length: 6514108999
date: Fri, 23 Jun 2023 07:06:20 GMT
last-modified: Thu, 08 Jun 2023 19:05:02 GMT
etag: "44ef1b51c0cc2200e29fed5cddbf8e27-408"
x-amz-storage-class: INTELLIGENT_TIERING
x-amz-server-side-encryption: AES256
x-amz-version-id: upVN9_QvGmQZMDWfVECqbytHgWjHTz4t
content-disposition: attachment; filename*=UTF-8''state_dict.bin; filename="state_dict.bin";
accept-ranges: bytes
server: AmazonS3
x-cache: Hit from cloudfront
via: 1.1 dff3fc94ddb54b32b708edf2668b23d2.cloudfront.net (CloudFront)
x-amz-cf-pop: SIN52-P1
x-amz-cf-id: JWTqKUDp8bVPD7Tt4DBySj8zbgT8G60sCA5BMC1cYF4vK9nx45f1Iw==
age: 200
vary: Origin
    • minorninth@lemmy.worldEnglish
      0·
      1 year ago

      That’s true, but I just checked a few CDNs and you won’t find one for less than $.01/GB. The lowest I found was $.03/GB.

        • omegastick@lemmy.mlOPEnglish
          3·
          1 year ago

          I was considering this. The hosting provider we use for model training runs doesn’t charge for ingress/egress. Their storage costs would eat us alive though haha. OVH looks much more promising.