Organization
Scale AI
Company that built the Remote Labor Index discussed as a test of AI performance on multi-day freelance tasks.
Stories on Angle
Background
Selected excerpts from Wikipedia. This is general background, not Angle's reporting, and it may have changed since the stories above were published.
Scale AI, Inc. is an American artificial intelligence infrastructure and software company based in San Francisco, California. Originally focused on data annotation, the company also offers RLHF services, large language model (LLM) evaluation, and enterprise software suites to build and deploy AI applications.
The company’s research arm, the Safety, Evaluation and Alignment Lab, focuses on evaluating and aligning LLMs. It also co-created the benchmark Humanity's Last Exam. Scale AI outsources data labeling through its subsidiaries, Remotasks, which focuses on computer vision and autonomous vehicles, and Outlier, which focuses on annotating data for LLMs.
Scale AI operates an LLM Red Team that conducts human adversarial testing to identify vulnerabilities, biases, and safety risks in AI models. The team has worked with OpenAI, Google DeepMind and national AI Safety Institutes to evaluate systems against complex threats, including cybersecurity vulnerabilities, jailbreaks, and agentic AI behaviors.
History
Scale was founded in 2016 by Alexandr Wang and Lucy Guo through Y Combinator. The pair had worked together at Quora. Initial investors of Scale included Dragoneer Investment Group, Tiger Global Management and Index Ventures. Guo was fired in 2018.
In August 2019, after Peter Thiel’s Founders Fund made a $100 million investment in Scale, the company's valuation exceeded $1 billion, and it acquired unicorn status.
Scale contracted with the United States Department of Defense in 2020.
In May 2021, Michael Kratsios, Chief Technology Officer of the United States under the Trump administration, joined as Scale AI's managing director and head of strategy.
By July 2021, Scale had reached a valuation of $7 billion, after a financing led by Greenoaks, Dragoneer Investment Group and Tiger Global Management. There was an increased demand for data labelling from clients in different industries.
In January 2022, Scale AI won a $250 million contract to give American federal agencies access to its suite of tools. In February 2022, Scale AI developed its Automated Damage Identification Service in response to the Russian invasion of Ukraine. Satellite imagery was analyzed to measure the damage to buildings, which were then geotagged and reported to humanitarian groups. In November 2022, Scale AI was recognized by Time on it’s Best Inventions of 2022 list. The company also opened an office in St. Louis in that same year.
In January 2023, Scale laid off 20% of its workforce.
In May 2023, Scale AI signed a deal with the US Army’s XVIII Airborne Corps, becoming the first AI company to deploy its LLM (known as Donovan) on a classified network.
In August 2023, Scale AI became OpenAI’s "preferred partner" to fine-tune GPT-3.5. The company's services were used to create ChatGPT. …
Remotasks
In 2017, Scale AI established Remotasks, a crowdworking platform to support the creation of labeled data for machine learning, particularly in areas such as computer vision and autonomous vehicles. The subsidiary has facilities in Southeast Asia and Africa.
In 2019, Scale AI set up a company called Smart Ecosystem Philippines to operate Remotasks within the country. In the Philippines, many of Remotasks' hires are freelance contractors not covered under labor laws. The pay for some annotation tasks dropped to less than one cent due to "vicious competition" after Remotask expanded to India as well as Venezuela. Late payments are reportedly "commonplace", and some workers received only a few percent of their promised compensation. In 2022, an Oxford Internet Institute study said Remotasks met the "minimum standards of fair work" in only one out of ten criteria.
Remotasks has been criticized for obscuring its affiliation with Scale AI, opaque communications, and abrupt changes in worker access in some regions. In early 2024, the platform terminated operations in several countries, including Kenya, Nigeria, and Pakistan, citing administrative and operational considerations.
Outlier
Outlier is a separate contributor platform operated by Scale AI, designed for generative AI data work, particularly in the development and fine-tuning of LLMs. Contributors on Outlier typically include professionals with advanced degrees, industry expertise, and native fluency in various languages. Outlier tasks involve content evaluation and reinforcement learning from human feedback (RLHF).
From the Wikipedia article Scale AI (revision of ). Wikipedia contributors; selected plain-text excerpts from this revision. Formatting, reference markers and non-prose material removed. Available under Creative Commons Attribution-Share Alike 4.0. Identity from Wikidata (CC0).