These are the news items I've curated in my monitoring of the API space that have some relevance to the API definition conversation and I wanted to include in my research. I'm using all of these links to better understand how the space is testing their APIs, going beyond just monitoring and understand the details of each request and response.15 Nov 2017
It has been a long time coming, but Twitter has finally started charging for premium access to their APIs. Until now, you could only access data via the free Twitter API with limitations, or pay to use Gnip at the enterprise level–nothing in between. I’ve long complained about how Twitter restricts access to our tweets, as well as the lack of transparency and honesty around their business model. I’ve complained so much about it, I eventually stopped writing about it, and I never thought I’d see the day where Twitter starts charging for access to their platform.
While I have concerns about Twitter further limiting access to our data by charging for API access, their initial release has some positive signs that give me hope that they are monetizing things in a sensible way. They seem to be focused on monetizing some of the key paint points around Twitter API consumption, like being able to get more access to historical data, offer more Tweets per request, higher rate limits, a counts endpoint that returns time-series counts of Tweets, more complex queries and metadata enrichments, such as expanded URLs and improved profile geo information. Twitter seems to be thinking about the primary pain we all experience at the lower rungs of Twitter access, and focusing on making the platform more scalable for companies of all shapes and sizes–which has been core to my complaints.
Twitter even provides a quote from a client which highlights what I’ve been complaining about for some time about the inequity in Twitter API access:
I wish these premium APIs were available during our first few years. As we grew, we quickly ran into data limitations that prevented expansion. Ultimately, we raised a round of funding in order to accelerate our growth with the enterprise APIs. With the premium APIs, we could have bootstrapped our business longer and scaled more efficiently. - Madeline Parra, CEO and Co-Founder, Twizoo (now part of Skyscanner) @TwizooSocial
This demonstrates for me how venture capital, and the way the belief systems around it railroad folks down a specific path, and is blind to those of us who do not choose the path of venture capital. Twitter’s new pricing page starts things off at $149.00 / month, after the current free public tier of access, and climbs up five tiers to $2,499.00 / month. Giving you more access with each access tier you climb. While not a perfect spacing of pricing tiers, something that sill might be difficult for some startups to climb, it is much better than what was there before, or should I say, what wasn’t there. At least you can scale your access now, in a sensible, above the board vertical way, and not horizontally with separate accounts. Incentivizing the more positive behavior Twitter should want to see via their API.
Twitter should have started doing this back in 2008 and 2009 to help throttle bad behavior on their platform. The platform would look very different today if there had been a level playing field in the API ecosystem, and developers weren’t forced to scale horizontally. API monetization and planning isn’t just about generating revenue to keep a platform up and running serving everyone. Sensible API monetization is about being aware of, and intimately understanding platform behavior and charging to restrict, shape, and incentivize the behavior you want to see on your platform. Twitter has missed out on a huge opportunity to truly understand the API community at this level, as well as generate much needed revenue over the years. Shutting out many good actors, incentivizing bad actors, and really only recognizing trusted partners, while being blind to everything else.
After a decade of complaining about Twitter’s practices in their ecosystem, and their clear lack of a business model around their API. I am happy to see movement in this area. While there is a HUGE amount of work ahead of them, I feel like monetization of the API ecosystem, and crafting of a sensible API plan framework is essential to the health and viability of the platform. It is how they will begin to get a hold on the automation that plagues the platform, and begin de-platforming the illness that has become synonymous with Twitter during the 2016 election, while also leveling the playing field for many of us bootstrapped startups who are trying to do interesting things with the Twitter API. I’ll keep an eye on the Twitter premium APIs, and see where things head, but for now I’m cautiously optimistic.
I’ve complained about unfair API pricing tiers several times over the last couple years, even declaring API access tiers irrelevant in a mult-API consumer world. Every time I write about this subject I get friends who push back on me that this is a requirement for them to generate revenue as a struggling startup. With no acknowledgement that their API consumers might also be struggling startups trying to scale consumption within these plans, only to reach a rung in the ladder they might not be able to actually reach. My goal in this storytelling isn’t to condemn API providers, but make them aware of what things look like from the other side, and that their argument essentially pulls up the ladder after they’ve gotten theirs–leaving the rest of us at the bottom.
My complaint isn’t with startups crafting pricing tiers, and trying to make their revenue projects more predictable. My complaint is when the plans are priced too far apart and I can’t afford to move from one plan to the next. More importantly, my complaint is when the tier I can’t moved from is rate limited with a cap on usage, and I can’t burst beyond my plans limits without scaling to the next access tier which I cannot afford to reach. I understand putting hard caps on public or free tier plans, but when you are squarely in a paid access tier, you shouldn’t be shut down when you reach the ceiling. Sure, I might pay a premium for each additional call, but I shouldn’t be shut down and forced to move to the next higher access tier–which might be out of my monthly price range. I just can’t go from $49.95 to $499.95 in monthly payments as a small business, sorry.
The key element that needs to be present for me, even in situations where I cannot afford to jump to the next tier, is the ability to go beyond my plans usage, with clear pricing regarding what I will pay. I may not be able to jump from $49.95 to $499.95 as monthly payments, but I might be able to burst, and absorb the costs as I need. If my plan is capped, and I cannot burst, and know what I will be charged for my usage (even if at a premium), it is a deal breaker for me. While I would prefer API providers do away with access tiers, and go with straight pay for what you use model (like Amazon), I accept the reality that API access plans help startups better predict and communicate their revenue. As long as they have this relief valve for each plan, allowing me to stay in my plan, but consume resources beyond what my tier allows.
Having relief valves for plans won’t hurt your revenue forecasting. I would argue it will actually help your revenue (not forecasting) with bursts in revenue that you wouldn’t see with just a capped plan approach. If you have trouble seeing API access in this way, I would argue that you are primarily an API provider, and building business exclusively focused on an exit. If you can empathize, I would say you are focused on delivering a set of services that people need, and you are probably an active consumer of other services, broadening your empathy spectrum beyond just a startup exit objectives. I honestly don’t want to mess with startups ability to generate revenue with this storytelling, I’m just trying to make the case for us startups on the API consumption side of the coin. Ok, I lied, I kind of do want to mess up the revenue strategy for startups who are exclusively focused on an exit, because when there isn’t a relief valve, you won’t find me signing up for one of your predictable plans in the first place.
I am learning about the AWS Marketplace through the lens of selling your API there, adding a new dimension to my API monetization and API plan research. I’ve invested a significant amount of energy to try and standardize what I learn from studying the pricing and plans for the API operations of the leading API providers. As I do this work I regularly hear from folks who like to tell me how I’ll never be able to standardize and normalize this, and that it is too big of a challenge to distill down. I agree that things seem too big to tame at the current moment, but with API pioneers like AWS, who have been doing this stuff for a while, you can begin to see the future of how this stuff will play itself out.
Amazon set into motion a significant portion of how we think about monetizing our API resources. The pay for what you use model has been popularized by Amazon, and continue to dominate conversations around how we generate revenue around our valuable digital assets. AWS has some of the most sophisticated pricing structure around their API services, as well as very mature pricing calculators, and have created markets around their resources (ie. spot instances for compute). You can see these concepts playing out in the guidance they offer software developers in their AWS Marketplace Seller Guide, which helps sellers modify their SaaS products to sell them through AWS Marketplace using two models: 1) metering, or 2) contract. When you list or application in the AWS Marketplace you must choose between one of these models, but both involve thinking critically about your monetization strategy, which includes your hard costs, as well as where the value will lie with your customers–striking the balance necessary to operate a viable API business.
According to the AWS Marketplace Seller Guide, each SaaS application owner listing through AWS Marketplace has two options for listing and billing your software:
- You can use the AWS Marketplace Metering Service to bill customers for SaaS application use. AWS Marketplace Metering Service provides a consumption monetization model in which customers are charged only for the number of resources they use in your application. The consumption model is similar to that for most AWS services. Customers pay as they go.
- You can use the AWS Contract Service to bill customers in advance for the use of your software. AWS Marketplace Contract Service provides an entitlement monetization model in which customers pay in advance for a certain amount of usage of your software. For example, you might sell your customer a certain amount of storage per month for a year, or a certain amount of end-user licenses for some amount of time.
No matter which plan you choose to deliver your API resources within, you will have to have the nuts and bolts of your API operations defined as part of your overall API monetization strategy. Each plan you offer needs to be derived from the hard costs involved with operations, but reflect the needs of your consumers. AWS gives you a handful of common dimensions for thinking through which type of plan you go with, and quantifying how you will be monetizing your API solution, in these nine areas:
- Users – One AWS customer can represent an organization with many internal users. Your SaaS application can meter for the number of users signed in or provisioned at a given hour. This category is appropriate for software in which a customer’s users connect to the software directly (for example, with customer-relationship management or business intelligence reporting).
- Hosts – Any server, node, instance, endpoint, or other part of a computing system. This category is appropriate for software that monitors or scans many customer-owned instances (for example, with performance or security monitoring). Your application can meter for the number of hosts scanned or provisioned in a given hour.
- Data – Storage or information, measured in MB, GB, or TB. This category is appropriate for software that manages stored data or processes data in batches. Your application can meter for the amount of data processed in a given hour or how much data is stored in a given hour.
- Bandwidth – Your application can bill customers for an allocation of bandwidth that your application provides, measured in Mbps or Gbps. This category is appropriate for content distribution or network interfaces. Your application can meter for the amount of bandwidth provisioned for a given hour or the highest amount of bandwidth consumed in a given hour.
- Request – Your application can bill customers for the number of requests they make. This category is appropriate for query-based or API-based solutions. Your application can meter for the number of requests made in a given hour.
- Tiers – Your application can bill customers for a bundle of features or for providing a suite of dimensions below a certain threshold. This is sometimes referred to as a feature pack. For example, you can bundle multiple features into a single tier of service, such as up to 30 days of data retention, 100 GB of storage, and 50 users. Any usage below this threshold is assigned a lower price as the standard tier. Any usage above this threshold is charged a higher price as the professional tier. Tier is always represented as an amount of time within the tier. This category is appropriate for products with multiple dimensions or support components. Your application should meter for the current quantity of usage in the given tier. This could be a single metering record (1) for the currently selected tier or feature pack.
- Units – Whereas each of the above is designed to be specific, the dimension of Unit is intended to be generic to permit greater flexibility in how you price your software. For example, an IoT product which integrates with device sensors can interpret dimension “Units” as “sensors”. Your application can also use units to make multiple dimensions available in a single product. For example, you could price by data and by hosts using Units as your dimension. With dimensions, any software product priced through the use of the Metering Service must specify either a single dimension or define up to eight dimensions, each with their own price.
These dimensions reflect the majority of software services being sold out there today. Make sure you not get stuck thinking about one way of thinking, like just paying per API call. Think about how your different API plans might have one or more dimensions, beyond any single use case.
- Single Dimension - This is the simplest pricing option. Customers pay a single price per resource unit per hour, regardless of size or volume (for example, $0.014 per user per hour, or $0.070 per host per hour).
- Multiple Dimensions – Use this pricing option for resources that vary by size or capacity. For example, for host monitoring, a different price could be set depending on the size of the host. Or, for user-based pricing, a different price could be set based on the type of user (admin, power user, and read-only user). Your service can be priced on up to eight dimensions. If you are using tier-based pricing, you should use one dimension for each tier.
Alll of these dimensions reflect the common building blocks of API plans and pricing which I’ve been tracking on for a number of years. It’s based upon Amazon selling their own APIs, as well as watching their customers price and sell their resources. Their pricing guide goes well beyond just APIs, and consider how you can generate revenue from any type of SaaS, but the dimensions they provide a place to start for ALL API providers, whether you are looking to sell them in the AWS Marketplace or not. You can find even more dimensions on my API plan research, but what Amazon provides will work for about 75% of the use cases out there today, and I’m looking to get you thinking critically your API monetization and plans, not provide you with too many options.
There just aren’t too many examples like this available, when it comes to thinking through how to price your APIs. My friends over at Algorithmia have pushed the conversation forward some with their algorithmic API marketplace, but you just don’t see this level of maturity with the pricing of resources over at Azure, Google, or others yet. Amazon is the furthest along in this journey. They have the most experience, and the most data regarding what digital resources are worth, and how you can measure and meter access. I think it will take a number of years to mature, but I think by 2020 we will see more standardization in how we structure the pricing for the most common digital resources available online–even if it is just the APIs we are selling on Amazon.
There will always be an infinite number of ways to charge for your APIs, but for many of the digital commodities that have become staples, we’ll see one or two common approaches stick. We’ll see less innovation in how we price the most used APIs, because those with market share will dictate the model that gets adopted, and others will emulate just so they can get a piece of the pie. As other API resources continue to mature, becoming digital commodities, we’ll see their pricing structure stabilize, and standardize to fit into market frameworks like we see emerging on the AWS platform. It will take time, but we’ll begin to see machine readable templates governing API pricing and plans, allowing cross platform markets to flourish, as API consumers figure out they can make API usage more predictable, budget-able, and orchestrate-able. We aren’t there yet, but you can see hints of this API economy over at AWS within their marketplace approach.
I am getting intimate with AWS API Gateway. Learning about what it does, and what it doesn’t do. The gateway brings a number of essential API management elements to the table, like issuing keys, establishing plans, and enforcing rate limits. However, it also lacks many of the other active elements of API management like billing for usage, which is an important aspect of managing API consumption for API providers. With AWS, things tend to work like legos, meaning many of their services work together to deliver a larger picture, and I’ve been learning more about how AWS API Gateway works with the AWS Marketplace to deliver some of the business of API features I’m looking for.
Here is the blurb from the AWS API Gateway documentation regarding how you can setup AWS API Gateway to work with AWS Marketplace, making your API available for sale as a SaaS service:
After you build, test, and deploy your API, you can package it in an API Gateway usage plan and sell the plan as a Software as a Service (SaaS) product through AWS Marketplace. API buyers subscribing to your product offering are billed by AWS Marketplace based on the number of requests made to the usage plan.<br /
To sell your API on AWS Marketplace, you must set up the sales channel to integrate AWS Marketplace with API Gateway. Generally speaking, this involves listing your product on AWS Marketplace, setting up an IAM role with appropriate policies to allow API Gateway to send usage metrics to AWS Marketplace, associating an AWS Marketplace product with an API Gateway usage plan, and associating an AWS Marketplace buyer with an API Gateway API key. Details are discussed in the following sections.<br /
To enable your customers to buy your product on AWS Marketplace, you must register your developer portal (an external application) with AWS Marketplace. The developer portal must handle the subscription requests that are redirected from the AWS Marketplace console.
While it is not exactly the billing model I’m looking for in my API management layer currently, it provides a compelling look at selling your APIs in a marketplace setting. There are a growing number of APIs available in the AWS marketplace, but still less than a hundred from what I can tell. Smells like an opportunity for API providers to step up and publish their APIs. The linkage between AWS API Gateway usage plan and AWS Marketplace product makes a lot of sense when you think about having an APIs as a product focus for your organization. I will be writing more about this relationship in the future, as I think it reflects how we should be thinking about our API service composition, and how we craft our API usage plans.
I’m going to setup one of my simple, more utility style APIs, like screen capture, or proper noun search, and deploy using AWS API Gateway, then publish as an AWS Marketplace product. I want to have a clear view of what it takes, and what the on-boarding process looks like for a consumer. Another thing I’m interested in doing, is how can I also offer a more wholesale version of these APIs. Meaning, I’m not metering their usage with AWS API Gateway and billing via AWS Marketplace. It will be a separate product that gets deployed in their AWS infrastructure as an EC2, RDS, and maybe AWS API Gateway, where they aren’t billed by usage, and billed by the implementation. This is a model I’ve been watching, considering, and thinking about for some time, but only now getting to where the public cloud infrastructure is able to support this type of API deployment and management. It will be interesting to see what I can make work.
I’m helping some clients think through their approach to API management. These projects have different needs, as well as different resources available to them, so I’m looking to distill things down to the essential components needed to get the job done. I spent some time thinking through the developer account basics, and now I want to break down the aspects of API consumption and usage around these APIs and developer accounts. I want to to think about the moving parts of how we measure, quantify, communicate, and invoice as part of the API management process.
Having A Plan We have developers signed up, with API keys that they’ll be required to pass in with each API call they make. The next portion of API management I want to map out for my clients is the understanding and management of how API consumers are using resources. One important concept that I find many API providers, and would be API providers don’t fully grasp, is service composition. Something that requires the presence of a variety of access tiers, or API plans, which define the business contract we are making with each API providers. API plans usually have these basic elements:
- plan id - A unique id for the plan.
- plan name - A name for the plan.
- plan description - A description for the plan.
- plan limits - Details about limits of the plan.
- plan timeframe - The timeframe for limits applied.
There can be more than one plan, and each plan can provide different types of access to different APIs. There might be plans for new users versus verified ones, as well as possibly partners. The why and how of each plan will vary from API provider to provider, but their are all essential to managing API usage. Something that needs to be well defined, in place, with APIs and consumers organized into their respective tiers. Once this is done, we can begin thinking about the next layer, logging.
Everything Is Logged Each call made to the API contains API key which identify the consumer, who should always be operating within a specific plan. Each API call is being logged as part of the web server, and ideally the API management layer, providing timestamped entries for every drop of APIs consumed. These logs are used across API management operations, providing a history of all usage that will be evaluated on a per API, as well as per API consumer level. If a request and response isn’t available in the API logs, then it didn’t happen.
Quantifying API Usage Every log entry recorded will have the keys for a specific API consumer, as well as the path of the API they are consuming. When this usage is viewed through the lens of the API plan an API consumer is operating within, you have the ability to quantify usage, and place a value on overall consumption by any time frame. This is the primary objective of API management, quantifying who is accessing API resources, and how much they are consuming. This is how valuable API resources are being secured, and in many situations monetized, using simple web APIs.
API usage begins with quantifying what consumers have used, but then there are other dimensions that should be considered as well. For example, usage across API users for a single path, or group of paths. Also possibly usage across APIs and consumers within a particular plan. Then you can begin to look across time periods, regions, and other relevant information, providing a more complete picture of how APIs are being put to use. This awareness is why API providers are employing API management techniques, beyond what can be extracted from database logs, or just website or application analytics–providing a more wholesale view of how APIs are consumed.
Invoicing For Consumption Now that we have all API usage defined, and available through the lens of API plans, we have the option of invoicing for consumption. We know each call API consumers have made, and we have a rate specific for each API as part of the API plans each consumer is subscribed to. All we have to do is the math, and generate an invoice for the designated time period (ie. daily, weekly, monthly, quarterly). Invoicing doesn’t have to always be settled with cash, as consumption may be internally, with partners, and with a variety of public consumers. I’d also warn against thinking of consumption always costing the consumer, and sometimes it might be relevant to pay API some consumers for some of their usage–incentivizing a particular type of behavior that benefits a platform.
Measuring, quantifying, and accounting for API consumption is the primary reason companies, organizations, institutions, and government agencies are implementing API management layers. It is how Amazon generates revenue from their Amazon Web Services. It is how Stripe, Twilio, Sendgrid, and other rockstar API providers generate the revenue behind their operations. It is how government agencies are understanding how the public is putting valuable public resources to work. API management is something that is baked into cloud platforms, with a variety of service providers available as well, providing a wealth of solutions for API providers to choose from.
Next I will be looking at the API account and usage layers of API management from the view of API provider, as well as the API consumer via their developer area. Ideally, API management solutions are providing the dashboards needed for both sides of this equation, but in some of the projects I’m working on this isn’t available. There is no ready to go dashboard for API providers or consumers to look at when standing up an AWS API Gateway in front of an API, falling short when it comes to going the distance as an API management solution. You can define accounts, issue keys, establish plans and limit manage API consumption, but we need AWS CloudFront, and other services to deliver on API logging, authentication, and other common aspect of management–with API management dashboards being a custom thing when employing AWS for management. One consideration for why you might go with a more robust API management solution, beyond what an API gateway offers.
I was doing some API security research and stumbled across vFeed, a “Correlated Vulnerability and Threat Intelligence Database Wrapper”, providing a JSON API of vulnerabilities from the vFeed database. The approach is a Python API, and not a web API, but I think provides an interesting blueprint for open source APIs. What I found interesting (somewhat) from the vFeed approach was the fact they provide an open source API, and database, but if you want a production version of the database with all the threat intelligence you have to pay for it.
I would say their technical and business approach needs a significant amount of work, but I think there is a workable version of it in there. First, I would create a Python, PHP, Node.js, Java, Go, Ruby version of the API, making sure it is a web API. Next, remove the production restriction on the database, allowing anyone to deploy a working edition, just minus all the threat data. There is a lot of value in there being an open source set of threat intelligence sharing databases and API. Then after that, get smarter about having a variety different free and paid data subscriptions, not just a single database–leverage the API presence.
You could also get smarter about how the database and API enables companies to share their threat data, plugging it into a larger network, making some of it free, and some of it paid–with revenue share all around. There should be a suite of open source threat information sharing databases and APIs, and a federated network of API implementations. Complete with a wealth of open data for folks to tap into and learn from, but also with some revenue generating opportunities throughout the long tail, helping companies fund aspects of their API security operations. Budget shortfalls are a big contributor to security incidents, and some revenue generating activity would be positive.
So, not a perfect model, but enough food for thought to warrant a half-assed blog post like this. Smells like an opportunity for someone out there. Threat information sharing is just one dimension of my API security research where I’m looking to evolve the narrative around how APIs can contribute to security in general. However, there is also an opportunity for enabling the sharing of API related security information, using APIs. Maybe also generating of revenue along the way, helping feed the development of tooling like this, maybe funding individual implementations and threat information nodes, or possibly even fund more storytelling around the concept of API security as well. ;-)
I’ve been playing with Tensor Flow for over a year now, specifically when it comes to working with images and video, but it has been something that has helped me understand what things looks like behind the algorithmic curtain that seems to be part of a growing number of tech marketing strategies right now. Part of this learning is exploring beyond Google’s approach, who is behind Tensor Flow, and understand what is going on at AWS, as well as Azure. I’m stil getting my feet wet learning about what Microsoft is up to with their platform, but I did notice one aspect of the Azure Machine Learning Studio emphasized developers to, “publish, share, monetize” their ML models. While I’m sure there will be a lot of useless vapor ware being sold within this realm, I’m simply seeing it as the next step in API monetization, and specifically the algorithmic evolution of being an API provider.
As the label says in the three ML models for sale in the picture, this is all experimental. Nobody knows what will actually work, or even what the market will bear. However, this is something APIs, and the business of APIs excel at. Making a digital resource available to consumers in a retail, or even wholesale way via marketplaces like Azure and AWS, then playing around with features, pricing, and other elements, until you find the sweet spot. This is how Amazon figured out the whole cloud computing game, and became the leader. It is how Twilio, Stipe and other API as a product companies figured out what developers needed, and what these markets would bear. This will play out in marketplaces like Azure and Google, as well as startup players like Algorithmia–which is where I’ve been cutting my teeth, and learning about ML.
The challenge for ML API entrepreneurs will be helping consumers understand what their models do, or do not do. I see it as an opportunity, because there will be endless amounts of vapor ware, ML voodoo, and smoke and mirrors trying to trick consumers into buying something, as well as endless traps when it comes to keeping them locked in. If you are actually doing something interesting with ML, and it actually provides value in the business world, and you provide clear, concise, no BS language about what it does–you are going to do well. The challenge for you will be getting found in the mountains of crap that is emerging, and differentiating yourself from the smoke and mirrors that we are already seeing so much of. Another challenge you’ll face is navigating the vendor platform course set up by AWS, Google, and Azure as they battle it out for dominance–a game that many of us little guys will have very little power to change or steer.
It is a game that I will keep a close eye on. I’m even pondering publishing a handful of image manipulation models I’ve been working on. IDK. I do not think they are quite ready, and I’m not even entirely sure they are something I want widely used. I’m kind of enjoying using them in my own work, providing me with images I can use in my storytelling. I don’t think the ROI is there yet in the ML API game, and I’ll probably just keep being a bystander, and analyst on the sideline until I see just the right opportunity, or develop just the right model I think will stand out. After seven years of doing API Evangelist I’m pretty good at seeing through the BS, and I’m thinking this experience is going to come in handy in this algorithmic evolution of the API universe, where the magic of AI and ML put so many people under their spell.
I remember when almost all the APIs out there gave us developers access to things we couldn’t ever possibly get on our own. Some of it was about the network effect with the early Amazon and eBay marketplaces, or Flickr and Delicious, and then Twitter and Facebook. Then what really brought it home was going beyond the network effect, and delivering resources that were completely out of our reach like maps of the world around us, (seemingly) infinitely scalable compute and storage, SMS, and credit card payments. In the early days it really seemed like APIs were all about giving us access to something that was out of our reach as startups, or individuals.
While this still does exist, it seems like many APIs have flipped the table and it is all about giving them access to our personal and business data in ways that used to be out of their reach. Machine learning APIs are using parlour tricks to get access to our internal systems and databases. Voice enablement, entertainment, and cameras are gaining access to our homes, what we watch and listen to, and are able to look into the dark corners of our personal lives. Tinder, Facebook, and other platforms know our deep dark secrets, our personal thoughts, and have access to our email and intimate conversations. The API promise seems to have changed along the way, and stopped being about giving us access, and is now about giving them access.
I know it has always been about money, but the early vision of APIs seemed more honest. It seemed more about selling a product or service that people needed, and was more straight up. Now it just seems like APIs are invasive. Being used to infiltrate our professional and business worlds through our mobile phones. It feels like people just want access to us, purely so they can mine us and make more money. You just don’t see many Flickrs, Google Maps, or Amazon EC2s anymore. The new features in mobile devices we carry around, and the ones we install in our home don’t really benefit us in new and amazing ways. They seem to offer just enough to get us to adopt them, and install in our life, so they can get access to yet another data point. Maybe it is just because everything has been done, or maybe it is because it has all been taken over by the money people, looking for the next big thing (for them).
Oh no! Kin is ranting again. No, I’m not. I’m actually feeling pretty grounded in my writing lately, I’m just finding it takes a lot more work to find interesting APIs. I have to sift through many more emails from folks telling me about their exploitative API, before I come across something interesting. I go through 30 vulnerabilities posts in my feeds, before I come across one creative story about something platform is doing. There are 55 posts about ICOs, before I find an interesting investment in a startup doing something that matters. I’m willing to admit that I’m a grumpy API Evangelist most of the time, but I feel really happy, content, and enjoying my research overall. I just feel like the space has lost its way with this big data thing, and are using APIs to become more about infiltrating and extraction, that it is about delivering something that actually gives developers access to something meaningful. I just think we can do better. Something has to give, or this won’t continue to be sustainable much longer.
Amazon Web Services recently updated their billing for EC2 instances to be by the second, which I really like, because I’ll fire up an instance and run for minutes, then shut things down. I’m just looking to process patent downloads, or other intensive workload projects. Beyond just EC2, the rest of Amazon’s platform is still very usage based. Meaning, you get charged for whatever you use, with unit pricing for each resource designed to compliment how it gets put to use. You get charged for the hard costs of compute, storage, and bandwidth, but you also see per message, job, entry, and other types of billing depending on the type of resource being delivered via API.
With this model for doing APIs, I’m wondering why so many API providers still have access plans and tiers. I’ve vented several times that I think service tiers are a legacy of a SaaS way of thinking and does not scale for API consumers. Maybe back when we used a handful of APIs, but the number of APIs I’m using is pushing 50 these days, and I can’t alway justify a monthly subscription to get what I need. I’m looking to just get access to valuable API resources, and get billed for whatever I use. If I don’t use anything for 6 months, I don’t get billed for anything. Also, I want to be able to run large jobs which consume intense amounts of resources without hitting tier and other limits–just charge me for what I use. If I have a $1,000.00 to spend today, let me spend it. Don’t make me jump through hoops.
I know the answer to my question regarding why so many API startups do this. It is because the resources being provided via the API isn’t the product, us API consumers are. They are looking to ensure a certain level of headcount, monthly, and annual subscribers, so that they can sell us to their investors, and ultimately whoever purchases us like cattle in the end. I’m sure there are other reasons for having pricing tiers, but this is still the primary reason we see SaaS based pricing continue to be so pervasive in an API driven world. For me, if an API provider has tiered pricing, I’ll almost always go look elsewhere. I just can’t manage 40 or 50 subscriptions, and if the number of APIs keep growing, I can’t handle 100-200 subscriptions. I just need to pay for the resources my business needs, and nothing more. I sure don’t have time to be a product in your startup cattle auction.
Pay as you go, usage based pricing is one way the cloud giants will suffocate out the small startups in the future. While startups are trying to court investors, and their acquirers, the cloud giants will just offer a competing service, drop the price to run competitors off, then once the coast is clear, raise prices back up to an profitable state. To compete, more API providers will have to go with a utility based pricing strategy, while also offering wholesale versions of their APIs in the marketplaces for AWS, Azure, and Google. I can’t help think that things might have been different if the scales didn’t tip so hard towards all of us API consumers being the product, and API providers focused more on running businesses, and catering to our real world business needs. Oh well, we got the world we have, I’ll just keep plowing forward.
I have been having more conversations with federal agencies as part of my work with my Skylight partners about API related microconsulting. One recent conversation, which I won’t mention the agency, because I haven’t gotten approval, involved bug bounties on top of an API they are rolling out. The agency isn’t looking for the regular technology procurement lifecycle around this project, they are just looking for a little bit of research and consulting to help ensure they are on the right track when it comes to hardening their API approach.
Micro consulting like this will usually not exceed $5,000.00 USD, and will always be a short term commitment. From my vantage point micro consulting will always be API related, and in this particular case involves studying how other API providers in the private sector are leveraging bug bounties to help harden their APIs either before they go public, or afterwards in an ongoing fashion. After I do the research I will be taking this work back to my team of consultants at Skylight, and we’ll put together formal report and presentation that we will bring back to the federal government agency to put into motion.
This approach to doing APIs in the federal government (or any government) is a win-win. It fits with my approach to doing research at API Evangelist, and it provides API expertise for federal agencies in small, affordable, and bite-size chunks. Government agencies do not have to wait months, or years, and spend massive amounts of money to gain access to API expertise. For Skylight, it gets our foot in the door within government, and helps demonstrate the expertise we bring to the table. Something that will almost always turn into additional micro procurement relationships, as well as potentially larger scale, ongoing project relationships.
Personally, I like my API consulting just like my APIs, small, and doing one thing well. I don’t like consulting contracts that try to do too much. I also like getting paid in chunks that can usually be put on the corporate credit card, and avoid too much purchase order, vendor system, 30, 60, and 90 day wrangling–or worse, not getting paid at all. You’ll hear me beating the micro consulting, and micro procurement drum a lot more in coming months. I’m going to be working to educate more government agencies, and even the enterprise of the potential when it comes to API related content creation, storytelling, training, and research. I’m predicting it will have the same effect as APIs are having on how companies, organizations, institutions, and government agencies are doing business in the digital economy.
API management was the first area of my research I started tracking on in 2010, and has been the seed for the 85+ areas of the API lifecycle I’m tracking on in 2017. It was a necessary vehicle for the API sector to move more mainstream, but in 2017 I’m feeling the concept is just too large, and the business of APIs has evolved enough that we should be focusing in on each aspect of API management on its own, and retire the concept entirely. I feel like at this point it will continue to confuse, and be abused, and that we can get more precise in what we are trying to accomplish, and better serve our customers along the way.
The main concepts of API management at play have historically been about authentication, service composition, logging, analytics, and billing. There are plenty of other elements that have often been lumped in there like portal, documentation, support, and other aspects, but securing, tracking, and generating revenue from a variety of APIs, and consumers has been center stage. I’d say that some of the positive aspects of the maturing and evolution of API manage include more of a focus on authentication, as well as the awareness introduced by logging and analytics. I’d say some areas that worry me is that security discussions often stop with API management, and we don’t seem to be having evolved conversations around service conversation, billing, and monetization of our API resources. You rarely see these things discussed when we talk about GraphQL, gRPC, evented architecture, data streaming, and other hot topics in the API sector.
I feel like the technology of APIs conversations have outpaced the business of APIs conversations as API management matured and moved forward. Advancements in logging, analytics, and reporting have definitely advanced, but understanding the value generated by providing different services to different consumers, seeing the cost associated with operations, and the value generated, then charging or even paying consumers involved in that value generation in real-time, seems to be being lost. We are getting better and the tech of making our digital bits more accessible, and moving them around, but we seemed to be losing the thread about quantifying the value, and associating revenue with it in real-time. I see this aspect of API management still occurring, I’m just not seeing the conversations around it move forward as fast as the other areas of API management.
API monetization and plans are two separate area of my research, and are something I’ll keep talking about. Alongside authentication, logging, analysis, and security. I think the reason we don’t hear more stories about API service composition and monetization is that a) companies see this as their secret sauce, and b) there aren’t service providers delivering in these areas exclusively, adding to the conversation. How to rate limit, craft API plans, set pricing at the service and tier levels are some of the most common questions I get. Partly because there isn’t enough conversation and resources to help people navigate, but also because there is insecurity, and skewed views of intellectual property and secret sauce. People in the API sector suck at sharing anything they view is their secret sauce, and with no service providers dedicated to API monetization, nobody is pumping the story machine (beyond me).
I’m feeling like I might be winding down my focus on API management, and focus in on the specific aspects of API management. I’ve been working on my API management guide over the summer, but I’m thinking I’ll abandon it. I might just focus on the specific aspects of conducting API management. IDK. Maybe I’ll still provide a 100K view for people, while introducing separate, much deeper looks at the elements that make up API management. I still have to worry about onboarding the folks who haven’t been around in the sector for the last ten years, and help them learn everything we all have learned along the way. I’m just feeling like the concept is a little dated, and is something that can start working against us in some of the conversations we are having about our API operations, where some important elements like security, and monetization can fall through the cracks.
People love to point out that APIs are unreliable. You can’t depend on them. They go away at any point, and they just aren’t something you want to be building your business on top of. When in reality APIs aren’t reliable it is the business, people, and investment behind them. The reality of the startup game is that us API consumers aren’t actually the customer, we are just numbers in a larger game where startup founders and their investors are looking for enterprise customers to purchase their startup getting the desired exit. The API is just about attracting consumers, who will do the legwork to bring in users, adding to the value of a company.
As the startup funding landscape continues to dry up, shift, and evolve towards more riskier and volatile versions of investment like ICOs, things are only going to get worse. Of course, few conversation will place the blame on the people and companies behind, but APIs will continue to be the scapegoat for the instability. It works just like the robots coming for your jobs. You never hear that rich people who own companies, that are making decisions to replace workers with robots are coming for your jobs. Its the robots. Technology in many forms makes for a great blame shield, absorbing the responsibility for the volatility, instability, and scams that are going on across the landscape.
In reality, nothing much changes for us API consumers. You need to get to know your API providers, well as the company and people behind them. Study their approach to operating their API. Do they communicate? Do they have proper support? Do they communicate their uptime status? What type of funding is propping them up, and the shape of their business model use. Make sure you always have a plan B if you can, and do not trust that ANY API will be around forever. If possible, come up with failover plans, and run drills with your team regarding what will happen when APIs fail, become unreliable, or go away entirely. Ultimately it is up to you to determine the impact unreliable APIs will have on your APIs. We can blame API providers, startups, and VCs as much as we want, but it is entirely up to us to help stabilize the API space for our users.
Investment, fundraising, and chasing exits will be the #1 area of instability for the API space. After that it will be the network, ranging from network neutrality going away to lack of investment in capacity and security. While the growth in the number of companies, organizations, institutions, and government agencies doing APIs will keep things on forward trajectory, the reputation of APIs will continue to take a hit when it comes to the stories people tell about who is to blame. APIs are the frontline for almost everything occurring online these days, and this will make it the first place everyone will be pointing the finger. It is up to us in the API community to keep telling stories putting the blame where it is deserved. If we are not telling stories, the other side will, and they’ll be firing up the pundits, continuing to let irresponsible API startups get away with their bullshit games.
Note: If my writing is a little dark this week, here is a little explainer–don’t worry, things will back to normal at API Evangelist soon.
I get regular emails from folks telling me how much they love what I do, then asking me to work for free. I’m totally happy for folks to inform me about their company, products, case studies, and other API goings on with their company. This is the bread and butter for my storytelling on API Evangelist–please keep it coming. However, the folks who ask me to work for free, what are you thinking? Where does this ethic come from in the tech sector, where folks expect you to work for their startup for free? It is something that has really gotten to ridiculous levels.
Join our webinar. Write a story for our blog. Contribute to a white paper. Join our podcast. Speak at our event. Teach a workshop. Come visit our company. Talk about our products and services. Help us craft our strategy. Do some research on this subject so we can use it. Tell us what we are doing wrong. Download, install, and play with our tool, and provide us with feedback. Take this 148 question survey that will only take you 45 minutes. Jump on the phone with us so we can pick your brain. Post our infographic to your site. Share this out via your Twitter account. Vote us up on Hacker News. Commit to our open source project. Help us create an API definition. Can you review this project specification. On, and on, and on. Oh, and hey I just wanted to email you again, because you didn’t respond to the last two emails!
Don’t get me wrong. If you know me, and we have history, feel free to ask me for favors, but remember, I do not have a job. I’m happy to help my friends, and love participating in podcasts with smart people, and reviewing my friends writing, because they scratch my back. They throw me bones, and have a history of given me exposure, and paying me for actual work. However, if we do not know each other, and have never talked, why on earth would you be asking me to work for free? Why would I want to take my brand that I’ve invested in building for seven years, and transfer that value to you for nothing? I don’t know you, your company, or anything about the value you deliver. I have to make a living just like you, and I can’t be working for free, or there would be nothing of value for both of us.
I know many of you are excited by the seemingly constant forward motion of the tech space, and that you might not realize that you are being rude. You may not be able to see it because you are in the middle of it, but much of startup culture is built on exploitation of others, and just because you are willing to be exploited by your investors, and those around you doesn’t mean you should expect others to fall in line. I’m a professional, and I’m happy to share my knowledge of the space, but it means you need to meet me halfway, and strike a business deal with me. This means emailing, DM’ing, and contacting me in a way that sets that table, not just expecting I will work for free. I appreciate your excitement, but please respect my time, and all the work I’ve put into my business, and I’ll do the same for you.
Note: If my writing is a little dark this week, here is a little explainer–don’t worry, things will back to normal at API Evangelist soon.
I’m really honored that some of my partners are kind enough to offer me a piece of the action in their companies, in exchange for what I do. I really am. However, going forward I’m going to have to decline any stock options in exchange for work, or advising, because it really doesn’t pencil out for me. I know it is the currency you are working with, getting investment in exchange for options, and trying to get knowledge, talent, and other forms of investment in exchange as well. I trust it will work out in your favor, but from my vantage point, there really is no upside in the game.
I regularly receive accusations of having and agenda because of real or perceived interest in companies, so with this hit on my brand, and the lack of return from the historic stock options I’ve had historically, it just doesn’t work out. My last advisor options netted me $319, for about 60 hours of work on, and travel costs out of my own pocket. I’m sure if I had more capital investment, and a bigger piece of the action, I’d fare better, but for the stake I get thrown–it just doesn’t do it for me. Plus, I have a manila folder in the filing cabinet of more significant shares that aren’t worth the paper they are printed on, so stock options really doesn’t float my boat in the first place.
Please don’t let this hurt your feelings. I know you are heavily invested in the value of your options, but I’m just not there. I’m happy to find other arrangements to make things happen, but with the way that stock options get wielded against me, and used to potentially discredit what I’m doing, it just isn’t worth anymore. As it stands today I only have stake in a single company–Skylight Digital (5%). I have NO stake in any API related company, and I am going to keep it that way. I’m not 100% against an equity stake, but the risk of low stake investments, and the high risk of damage to my brand leaves me extremely critical about any conversation in this area.
It is another fascinating area of the startup game in my view, where people like punching down, instead of up when it comes to questioning the ethics and credibility folks in the API space. People be all questioning my ethics, but afraid to ever go after startups, or investors, because it might hurt their chances to be invited into the club some day! It’s a crazy financing reality in the API space, where up is down, and down is up, and I’m just beating that API Evangelist drum. #onward
Note: If my writing is a little dark this week, here is a little explainer–don’t worry, things will back to normal at API Evangelist soon.
I am the first to admit that I suck at the money game. I just don’t care. Don’t get me wrong, I’ve made a significant amount of money in my career, and command a phat six figure salary when I’ve done the job thingy, and I don’t have a problem asking for a decent rate when I’m consulting. It is just that I don’t care about climbing the money ladder, because I realized in my early 20s that it was never enough. I’ve had business partners run off with hundreds of thousands of dollars in cash, screw me out of the equity in multiple companies, and I’ve experienced what people will do to get at that next rung of the ladder. I quickly saw that you are never happier with each rung you climb, and often times you end up much unhappier the higher up you go–which is why I stay where I am at.
If you have been one of my partners you know I suck at invoicing regularly. Honestly, I don’t think about it until I have to. Which regularly I forget an entire month. Unless I’m looking looking at eviction, or have something I need to purchase, I’m more obsessed with working. I love what I do (mostly), and if someone would just drop money in my account each month, I just keep beating the API Evangelist drum, and never look to make beyond what it takes to keep me going each month. Being independent is a hustle, and it takes the support of my partners (thanks 3Scale, Runscope, Restlet, and Tyk) to make this work, along with some side consulting hustle to fill in the gaps. I mostly prefer the low monthly sponsorship money because the consulting hustle is a chore, and chasing down the money can be a full time job. I’d say 1/3 of the time its about getting plugged into corporate system(s), 1/3 of the time smooth sailing, and 1/3 of the time I’m getting screwed.
Honestly, the companies who want an NDA, contract, and all the trimmings are the ones I’ll usually walk away from. When I have a contract, I almost always get screwed over. I have more unpaid contracts from more well known corporations and startups than I can shake a stick at. I’m amazed at the folks who talk me up, tell me how much they love what I do, and want my services…then get what they need, don’t pay their invoice, and go radio silent. I also get many folks who tell me how their API budget wasn’t as big as they had hoped this quarter, and my fees would be such a big hit. Another I hear regularly is startups waiting for their investment, and at some point they’ll kick me down when they get theirs. The space is full of excuses regarding how hard-done-by everyone is with their operating budgets, runway, and what not. Yeah, I know what you mean–your hard times means I won’t make rent this month. I get it. I do.
This is all a game. The only difference is I’m admitting it, and you aren’t. I feel like we are all in this together, trying to build community, and you are secretively in it for yourself, and working it from every angle. Don’t worry, I know not all of you are like this, and when I’m done with API rant week, I’ll showcase some of the heroes in my world (you know who you are). I just wanted to take the time to shame some of you for the greedy, inconsiderate individuals you are, playing a nice game on the front, but making sure you always get yours. The differences is that you’ll have a rough time in the next meeting, and I’ll be out of my apartment, unable to make it to my next consulting gig, or just unable to meet my wider personal and professional obligations. But it’s cool, I’m sure you got this all figured out, and you will sleep well at night. It’s just business.
You are in a sweet spot. You got a fat six figure job in the coolest department of your company, building out your API platform. You have a decent budget (never as much as you want) to throw hackathons, run Google and Twitter ads, and you can buy schwag to give away at your events. Sure there is a lot of pressure to deliver, but you are doing pretty well. All you gotta do is convince 3rd party developers to do thing with your companies APIs, develop web, mobile, voice, and other applications that generate buzz and deliver the return on investment your bosses are looking for.
It is all about you and your team. Let’s get to work growth hacking! Attract as may new users as we can, and convince them to build as much as we possibly can. Let’s get them to develop SDKs, write articles for us on their blog, speak at our events, favorite things on hacker news, and whatever activities that we can. Your objective is to extract as much value from your API operations as you possibly can, and give nothing back. Expect developers to work for free. Expect your hackathons attendees to come up with the next great idea, build it, and hand it over to you for very little in return. This isn’t a partnership, this is an API ecosystem, and your team is determined to win at all costs.
Your API isn’t a two-way street. All roads lead to your success, and your bosses getting what they want. You don’t care that 3rd party developers should be compensated, or that they have any rights to their intellectual property. The 5% of them that successfully build applications, we will offer them a job in exchange for it, or we’ll just replicate it internally, decrease their rate limits, and increase their error rates so that they can’t compete. Sure you want people to still feel inspired, but not enough so that they’ll ever be able to sustain their applications–the only sustainable application around here will be owned by the platform. After all, this is all just business–it is nothing personal.
I’ve been looking at my human services API work through a microservices lens, triggered by the deployment of a reduced functionality version of the human services implementation I was working on this week. I’m thinking a lot about the technical side of decoupling services using APIs, but I want to also take a moment and think about the business side of decomposing services, while also making sure they are deployed in a way that meets both the provider and consumer side of the business equation. My human services microservice implementation is in the public service, which is a space where the business conversation often seems to disappear behind closed doors, but in reality needs to be front and center with each investment (commit) made into any service.
Let’s take a moment to look at the monetization strategy and operational plan for my human service microservice. Yes, a public data microservice should have a monetization strategy and plan for operating and remaining sustainable. The goals for this type of microservice will be radically different than it would be for a commercial microservice, but it should have one all the same.
- Monetization - How am I evaluating the investment into this project alongside any value that is generated, which I can potentially capture or exchange some value for some money to keep going.
- Acquisition - What did it take for me to acquire the data and skills necessary to make the deployment possible.
- Development - What time was invested in setting up the platform, developing the schema, data, definitions, code, and visual elements.
- Operations - What does it take to operate the service? Maintain it, answer questions, provide support, and other realities of providing an online service today.
- Direct Value - What are the direct benefits of having this service available to people looking for human services, or organizations looking to provide human services.
- Indirect Value - What are the indirect benefits of having this service available, like increased conversation around human services, or maybe traffic and awareness of the Open Referral organization.
- Partners - What partnership opportunities are actively being sought out, or will be opened up by having this service available.
- Reporting - What type of reporting is necessary to operate and monetize this service, from tracking page views to understanding who is integrated with the data, and consuming data via APIs, or possibly through continuous integration of the Github repository.
- Plan - What is my plan for making this service available to the public, partners, or maybe internally use across my own projects.
- Elements - My human services location API is designed to be publicly available, forkable, and reusable by anyone.
- Limits - There are no limits on how each human services microservice can be used, or forked and reused. Ideally, any implementation provides attribution, and acknowledges the source of framework or data, but there really are no rules.
- Metrics - I am measuring unique page views on each page, including of the machine readable YAML, JSON, and other formats. I’m also tracking on stars, forks, and commits for each service.
- Commercial - Providing a clear track for commercial vendors to understand that the project needs their support, and can be improved upon, and evolved through their underwriting and support.
I need to have a coherent snapshot of what I’ve invested into each of my service. I need to have a base plan for how I will be executing the business side of this service–even if it just making something available for free. There are two dimensions to this conversation: 1) My view as the creator of this service 2) The view of folks who fork and implement as a service. Both dimensions should have a monetization snapshot, and a plan for executing within this business snapshot. I need all of this decoupled from any other service I am offering, but ideally they all use a common set of reusable patterns, just like the technical aspects of my microservices effort.
Just like needing the compute, database, DNS, and other technical layers to be stable and scalable across my microservices, I need the costs associated with these elements predictable and affordable. I need to know what it costs to define, design, deploy, manage, and deprecate my services. I need to know the best path forward for making them public, keeping them private, and being honest with commercial partners about the value that is being generated, both directly and indirectly. I need a way to report my costs, and revenue across hundreds or thousands of these services. I need to be able to scale this both horizontally across many services, but also vertically for single services which get deployed over and over, reused and continuously integrate wherever they are needed using Github.
I’ll keep applying this model across my human services project. I’m thinking that I’m going to be developing a whole buffet of human service microservices that run a 100% on Github like this, but I am also playing around with other varietals that run on other popular cloud platforms as well. I’m not one to subscribe to any particular API dogma, I’m just looking to explore what is possible, and do all of it as cheaply as I possibly scan, growing the number of projects I’m able to tackle. Of course, none of this would be possible without my partners funding my work, and helping me connect the technical and business aspects of doing API Evangelist.
I was reading about the difficulties the City of New York was having when it comes to migrating off of the Palantir platfom, while also reading about the latest cybersecurity drama involving ransomware. I’m spending a lot of time studying cybersecurity lately, partly because they involve APIs, but mostly because it is something that is impacting every aspect of our lives, including our democracy, education, and healthcare. One thing I notice on the cybersecurity stage, is that everything is a much more extreme, intense, representation of what is going on in the mainstream tech industry.
Ransomware is software that gets installed on your desktop or servers and locks up all your data until you pay the software developer (implementor) a ransom. Ransomware is just a much faster moving version of what many of us in the software industry call vendor lock-in. This is what you are seeing with Palantir, and the City of New York. What tech companies do is get you to install their software on your desktop or servers, or convince you to upload all your data into the cloud, and use their software. This is business 101 in the tech industry. You either develop cloud-based software, something that runs on-premise, or you are a mix of both. Ideally, your customers become dependent on you, and they keep paying your monthly, quarterly, or annual subscriptions (cough cough ransom).
Here is where the crafty API part of the scheme comes in. Software providers can also make APIs that allow your desktop and server to integrate with their cloud solutions, allowing for much deeper integration of data, content, and algorithms. The presence of APIs SHOULD also mean that you can more easily get your data, content, and algorithms back, or have kept in sync the whole time, so that when you are ready to move on, you don’t have a problem getting your data and content back. The problem is, that APIs “CAN” enable this, but in many situations providers do not actually give you complete access to your data, content, or algorithms via API, and enable the true data portability and sync features you need to continue doing business without them.
This is vendor lock-in. It is a much friendlier, slower moving version of ransomware. As a software vendor you want your software to be baked in to a customers operations so they are dependent on you. How aggressively you pursue this, and how much you limit data portability, and interoperability, dictates whether you are just doing business as usual, or engaging in vendor lock-in. One thing I’m hopeful for in all of this, are the vendors who see transparency, observability, interoperability, and portability of not just the technical, but also the business and politics of delivering technology as a legitimate competitive advantage. This means that they will always be able to out maneuver, and stay ahead of software vendors who practice vendor lock-in and ransomware, whether of the slow or fast moving variety.
The General Services Administration(GSA) has an API strategy, which describes “GSA’s future direction for agencywide API management including design, development, architecture, operations, and support, and security.” Ok, let’s pause there. I want to point out that this isn’t just an API design guide. That is a portion of it, but it also touches on some of the most obvious (deployment), and the most critical aspects (security) of API operation–important stuff.
The objectives for the GSA crafting an API strategy are:
* Harness API management to maximize customer value and technical efficiencies. * Adopt industry best practices with API design, development, and management. * Consider opportunities for revenue through API offerings.
I always enjoy seeing federal agencies talk about efficiencies and best practices, but it gives me hope that all of this might actually work when I see federal agencies actually “considering opportunities for revenue through API offerings”. Especially at the GSA, who provides technical guidance to the other 400+ federal agencies, as well as at the state and municipal level. I am not excited about our government charging money for digital resources, I am excited about our government exploring how it will generate revenue to sustain itself in the future.
I know there are a lot of open data advocates who can’t wrap their mind around this, but this is how the government will generate needed tax revenue to operate in the future–using APIs. Our government currently generates a portion of its revenue from the sale of physical resources like timber, mining, and agriculture, why should things be different when it comes to taxing and regulating digital resources being made available via the web, mobile, and device applications. While there is still lots to figure out on this front, I am glad to see the GSA putting some thought into the role APIs will play in the future of funding the government services we all depend on each day.
I was having a discussion with an investor today about the potential of algorithmic-centered API marketplaces. I’m not talking about API marketplaces like Mashape, I’m more talking about ML API marketplaces like Algorithmia. This conversation spans multiple areas of my API lifecycle research, so I wanted to explore my thoughts on the subject some more.
I really do not get excited about API marketplaces when you think just about API discovery–how do I find an API? We need solutions in this area, but I feel good implementations will immediately move from useful to commodity, with companies like Amazon already pushing this towards a reality.
There are a handful of key factors for determining who ultimately wins the API Machine Learning (ML) marketplace game:
- Always Modular - Everything has to be decoupled and deliver micro value. Vendors will be tempted to build in dependency and emphasize relationships and partnerships, but the smaller and more modular will always win out.
- Easy Multi-Cloud - Whatever is available in a marketplace has to be available on all major platforms. Even if the marketplace is AWS, each unit of compute has to be transferrable to Google or Azure cloud without ANY friction.
- Enterprise Ready - The biggest failure of API marketplaces has always been being public. On-premise and private cloud API ML marketplaces will always be more successful that their public counterparts. The marketplace that caters to the enterprise will do well.
- Financial Engine - The key to markets are their financial engines. This is one area AWS is way ahead of the game, with their approach to monetizing digital bits, and their sophisticated market creating pricing calculators for estimating and predicting costs gives them a significant advantage. Whichever marketplaces allows for innovation at the financial engine level will win.
- Definition Driven - Marketplaces of the future will have to be definition driven. Everything has to have a YAML or JSON definition, from the API interface, and schema defining inputs and outputs, to the pricing, licensing, TOS, and SLA. The technology, business, and politics of the marketplace needs to be defined in a machine-readable way that can be measured, exchanged, and syndicated as needed.
Google has inroads into this realm with their GSuite Marketplace, and Play Marketplaces, but it feels more fragmented than Azure and AWS approaches. None of them are as far along as Algorithmia when it comes to specifically ML focused APIs. In coming months I will invest more time into mapping out what is available via marketplaces, trying to better understand their contents–whether application, SaaS, and data, content, or algorithmic API.
I feel like many marketplace conversations often get lost in the discovery layer. In my opinion, there are many other contributing factors beyond just finding things. I talked about the retail and wholesale economics of Algorithmia’s approach back in January, and I continue to think the economic engine will be one of the biggest factors in any API ML marketplace success–how it allows marketplace vendors to understand, experiment, and scale the revenue part of things without giving up to big of a slice of the pie.
Beyond revenue, the modularity and portability will be equally important as the financial engine, providing vital relief valves for some of the classic silo and walled garden effects we’ve seen the impact the success of previous marketplace efforts. I’ll keep studying the approach of smaller providers like Algorithmia, as well as those of the cloud giants, and see where all of this goes. It is natural to default to AWS lead when it comes to the cloud, but I’m continually impressed with what I’m seeing out of Azure, as well as feel that Google has a significant advantage when it comes to TensorFlow, as well as their overall public API experience–we will see.
I am going through my entire infrastructure lately, quantifying the products and services that API Evangelist offers, and the partnerships that make everything go round. As I do in my work as the API Evangelist, I'm looking to work through my thoughts here on the blog, and this week I have an interesting topic on the workbench--the API Evangelist remarketing tags.
According to Google, remarketing tags are: "To show ads to people who have visited your desktop or mobile website, add the remarketing tag to your website. The tag is a short snippet of code that adds your website visitors to remarketing lists; you can then target these lists with your ads. If your website has a Google Analytics tag, you can use this tag instead and skip adding the AdWords remarketing tag."
Over the last couple of years, I've had 3Scale's remarketing tags on my site. 3Scale has been my number one supporter for the last three years, and API Evangelist wouldn't even be a thing without them. They have provided me with financial support each month for a variety of reasons. We've never fully itemized what is in exchange for this support, primarily it has been to just invest in me being API Evangelist and help 3Scale be successful, but one of the things on the table for the last couple of years has been that I published the 3Scale remarketing tags on my network of sites.
So, if you visited API Evangelist in the last three years, it is likely you saw 3Scale ads wherever you went on the web. As I'm evaluating all the products and services I offer, quantify my partnerships, and identify the value that API Evangelist brings to the table, I find myself thinking deeply about this practice. Should I be selling my visitors data like this? If you visit API Evangelist, should I allow my partners to target you with advertising on other sites you visit? As I think about the challenges I face in accepting money from startups, I'm forced to look at myself in the mirror as I think about selling remarketing tag access on my website(s).
Ok, I take the high road. I don't sell remarketing tags to my partners. I even take the Google tracking off my website. I'm happy with tracking my traffic at the DNS level, I do not need to do this. Except for the fact that it brings in revenue, I need to pay my server bills, rent, and everything else that keeps me doing API Evangelist. It isn't easy making a living being independent in the tech space, and I need every little bit of revenue I possibly can. It's not an insignificant amount of money either. I make a couple thousand dollars a month this way, as the traffic to my website is pretty high quality, at least in the eyes of an API service provider looking to sell you their warez.
I am still thinking on this one. I need to make a decision this month regarding what I am going to do with the remarketing tags for API Evangelist. Do I want to sell them entirely to one partner? Do I want to sell individual ones for each area of my research, like design, deployment, management, monitoring, and the other 70 areas of the API lifecycle I track on? IDK. It is a shitty place to be, with so few options for monetization of my work, beyond just the current advertising industrial complex (AIC)™. I'd love to hear your thoughts, either publicly or privately. It's a tough decision for me because if I choose to not do it, I risk API Evangelist going away, and me having to go get a job. *shudder* *shudder again* I'm exploring other ways of generating revenue, things like creating classes, and other content I can sell access to, so hopefully I can just walk away from remarketing tags without it killing me off. IDK WTF
It is a hustle to do API Evangelist. I've been lucky to have the support of 3Scale since 2013, without them API Evangelist would not have survived. I'm also thankful for the community stepping up last year to keep the site up and running, keeping it community focused thing, and not just yet another vendor mouthpiece. I make my money providing four ad slots on the site, by selling guides and white papers, and by consulting and creating content for others. It is a hustle that I enjoy much more than having a regular job, even though it is often more precarious, and unpredictable regarding what the future might hold.
Taking money from companies always creates a paradox for me. People read my stories because they tend to be vendor neutral and focus on ideas, and usable API-centric concepts. While I do write about specific companies, products, services, and tooling, I primarily try to talk about the solutions they provide, the ideas and stories behind them, steering clear of just being a cheerleader for specific vendor solutions. It's hard, and something I'm not always successful at, but I have primarily defined my brand by sticking to this theory.
This approach is at odds with what most people want to give me money for. 3Scale has long supported me and invested in me being me, doing what I do--which is rare. Most companies just want me to write about them, even if they understand the API Evangelist brand. They are giving me money, and in exchange, I should write about them, and their products and services. They often have no regard to the fact that this will hurt my brand, and run my readers off, falling short of actually achieving what they are wanting to achieve. I get the desire to advertise and market your warez, but the ability for companies to be their own worst enemy in this process is always fascinating to me.
I get regular waves of folks who want to give me money. I'm talking with and fending off a couple at the moment. So I wanted to think through this paradox (again), and talk about it out loud. It just doesn't make sense for me to take money from a company to write blog posts, white papers, and guides about their products and services. However, I feel like I can take money from companies to write blog posts, white papers, and guides about an industry or topic related to the problems they provide solutions for, or possesses a significant audience that might be interested in their products and services. I consider this underwriting and sponsorship of my API Evangelist research, where they receive branding, references, and other exposure opportunities along the way.
Ideally, all branding, reference, and exposure elements are measurable through the tracking of impressions and links. What was the reach? What is the scope of exposure? It is difficult to sustain any relationship without measuring success and both parties are unable to articulate and justify the financial underwriting and support. In some cases, I get paid a finder's fee for referrals, but this can be very difficult to track on and validate--something that requires a company to be pretty ethical, and truly invested in partnerships with smaller brands like me. I prefer to rely on this, as opposed to any sort of affiliate or reseller style tracking systems. I like companies that ask their customers, "how did you learn about us?", as they tend to actually care about their employees, partners, other stakeholders at a higher level.
Sometimes I take an advisor, or even technology investor role in startups, taking on a larger stake in outcomes, but these are something that is very rare. I have a manila file folder filled with stock options, and stakes I have in companies that never reached an exit, or when they did I was written out of things--when I do get a check from startup founders, I'm always floored. This does happen, but is truly a rare occurrence. I wish there were more decoupled, plug and play opportunities to invest in startups as an advisor, researcher, analyst, and storyteller, but alas the system isn't really set up for this type of thinking--people prefer the big money plays, over smaller, more meaningful contributions--it's about the money man.
Anyways, every time I visit this conversation in my head I come back to the same place. I'm happy to take money from companies to generate short form and long form content about an industry or topic. If there are finder fees for referrals, great! I leave it up to you to track on and come back to me with the details, and any specific event--while I will stay focused on what I do best, the thing you are investing in me to do. I'm mildly interested in opportunities to become more invested in your companies overall success, honestly, I just don't trust that the system will deliver in this area, and is more often just looking to extract value from me. I have seen too much in my 30-year career. However, I always welcome folks who want to prove me wrong! ;-)
In the end, my mission stays the same. I'm interested in studying where the API space has been, where it is at, and where it might be going. Then, I am interested in telling stories from what I learn in this process. If you want to invest in any of this, let me know. I could use your help.
I get why SaaS, and API providers offer a handful of pricing plans and tiers for their platforms, but it isn't something I personally care for as an API consumer. I've studied thousands of plans and pricing for API providers, and have to regularly navigate 50+ plans for my own API operations, and I just prefer having access to a wide range of API resources, across many different companies, with a variety of usage limitations and pricing based upon each individual resources. I really am getting tired of having to choose between bronze, gold, or platinum, and often getting priced out completely because I can scale to the next tier as a user.
I understand that companies like putting users into buckets, something that makes revenue predictable from month to month, or year to year, but as we consumer more APIs from many different providers, it would help reduce the complexity for us API consumers if you flattened the landscape. I really don't want to have to learn the difference between each of my provider's tiers. I just want access to THAT resource via an API, at a fair price--something that scales infinitely if at all possible (I want it all). Ultimately, I do not feel like API plans and tiers will scale to API economy levels. I think as API providers, we are still being pretty self-centered, and thinking about pricing as we see it, and we need to open up and think about how our API consumers will view us in a growing landscape of service providers--otherwise, someone else will.
As I pick up my earlier API pricing work, which has two distinct components: 1) all API resources and pricing available for a platform 2) the details of plans and tiers which a complex list of resources, features, and pricing fit into. It would be much easier to just track resources, the features they have, and the unit price available for each API. Then we could let volume, time-based agreements, and other aspects of the API contract help us quantify the business of APIs, without limiting things to just a handful of API contract plans and tiers, expanding the ways we can do business using APIs.
As an API provider, I get that a SaaS model has worked well to quantify predictable revenue in a way that makes sense to consumers, but after a decade or more, as we move into a more serverless, microservices, devops world, it seems like we should be thinking in a more modular way when it comes to the business of our APIs. I'm sure all you bean counters can get out of your comfort zone for a bit, and change up how you quantify access to your API resources, following the lead of API pioneers like Amazon, and just provide a master list of API, CLI, and console resources available for a competitive price.
I think way too much about the digital bits being transmitted online each day. I study the APIs that are increasingly being used to share these bits via websites, mobile, and other Internet-connected devices. These bits can be as simple as your messages and images or can be as complex as the inputs and outputs of algorithms used in self-driving cars. I think about bits at the level up from just the 1s and 0s, at the point where they start to become something more meaningful, and tangible--as they are sent and received via the Internet, using web technology.
The average person takes these digital bits for granted, and are not burdened with the technical, business, and political concerns surrounding each of these bits. For many other folks across a variety of sectors, these bits are valuable and they are looking to get access to as many of them as you can. These folks might work at technology startups, hedge funds, maybe in law enforcement or just tech-savvy hacker or activist on the Internet. If you hang out in these circles, data is often the new oil, and you are looking to get your hands on as much of it as you can, and are eager to mine it everywhere you possibly can.
In 2010, I started mapping out this layer of the web that was emerging, where bits were beginning to be sent and received via mobile devices, expanding the opportunity to make money from these increasingly valuable bits on the web. This move to mobile added a new dimension to each bit, making it even more valuable than they were before--it now possessed a latitude and longitude, telling us where it originated. Soon, this approach to sending and receiving digital bits spread to other Internet-connected devices beyond just our mobile phones, like our automobiles, home thermostats, and even wearables--to name just a few of the emerging areas.
The value of these bits will vary from situation to situation, with much of the value lying in the view of whoever is looking to acquire it. The value of a Facebook wall post is worth a different amount to an advertiser looking for a potential audience, then it will be to law enforcement looking for clues in an investigation, and let's not forget the value of this bit to the person who is posting it, or maybe their friends who are viewing it. When it comes to business in 2017, it is clear that our digital bits are valuable, even if much of this value is purely based on perception and very little tangible value in the real world. With many wild claims about the value and benefit of gathering, storing, and selling bits.
Markets are continually working to define the value of bits at a macro level, with many technology companies dominating the list, and APIs are defining the value of bits at the micro level--this is where I pay attention to things, at the individual API transaction level. I enjoy studying the value of individual bits, not because I want to make money off of them, but because I want to understand how those in power positions perceive the value of our bits and are buying and selling our bits at scale. Whether it is compute and storage in the cloud, or the television programs we stream, and pictures and videos we share in our homes, these bits are increasing in value, and I want to understand the process how everything we do is being reduced to a transaction.
When thinking about generating revenue generated from APIs it is easy to focus on directly charging for any digital resource being made available via the API. If it's an image, we charge per API call, and maybe the amount of MB transferred. If it's messaging, we charge per message. There are plenty of existing examples out there regarding how you directly charge for data, content, or algorithms using APIs, and an API way of doing business--look to Amazon, Twilio, and other pioneers.
Where there are fewer examples and less open discussions, is around the value of the operation level of APIs,Â and making these data available via APIs--yes APIs for APIs. Modern approaches to doing APIs are all about requiring each application to use an API key with each call they make, the logging of each request and response, possessing the identifying key for each application. This is how API providers are developing an awareness of who is accessing resources, how they are being put them to use, and specific details about each application, and maybe even the users involved.
Sometimes the value generated at this layer doesn't exist. Due to restrictive access models, and direct revenue models, there isn't much going on operationally, so there isn't much value generated. However, when there is heavy usage around APIs, the exhaust of the API management layer can become increasingly valuable. What are people searching for? What are applications most popular? Which geographic regions are the most active? There is a pretty lengthy laundry list of valuable data points being applied across modern API operations, that are helping API providers better understand what is going on, that aren't often being included as part of the API road map, and future revenue models.
Ok, let me pause here for a moment. I identify the value being generated at this layer because I see existing providers reaching this realization in their operations, as well as wanting to help other providers see the potential being achieved by successful API providers. I also acknowledge there is a lot of exploitation, abuse, and surveillance going on at this level, which is one of the biggest reasons I'm encouraging more transparency, observability, and discussion about this layer. I want API providers to understand the potential, but I also want API consumers and the end users of their applications to understanding what is going on at the API operational layer as well.Â
The current model I'm looking at through this lens currently is around my Open Referral Human Services Data Specification (HSDS) work, where I'm trying to help define the operational layer of human services APIs, as well as the direct API access to this critical public data. I am asking the question of how stewards of this very important data at the municipal level leverage APIs to make their valuable resources more accessible, and put to work where it is most needed, while also being able to generate and harness the valuable particles generated as part of an API exhaust system. What are people searching for? How are demographics evolving in a city, and how can city services shift and evolve too. Making the operational layer available via API so that it is available to key decision makers, even if those are private sector decisions makers who are required to pay for access to this intelligence--bringing critical new revenue streams for data stewards.
Let's pause again and be honest about the privacy concerns here. Access at this layer needs an additional level of scrutiny and care, over the direct access layers. Examples I'm concerned with can be seen in searches for Muslim religious services, or possibly LGBTQ services, and other information that could be used to violate the privacy and rights of application developers and end users. There are numerous privacy and security concerns at this level, but the inverse of these concerns also highlight the value of a data access exhaust system at this level. This is important information, that can provide some real time signals for both the public and private sector to consider more deeply.
I am purposely using the word exhaust here, specifically as a noun, as I don't think people are considering this layer, and may often see log files, and other ways data being generated in this as way as an unusable byproduct and output of web and mobile operations. I want to help people see the potential dangers of exhaust from API-centric data operations, but also understand that when it is captured, it can become valuable, similar to natural gas capture from recycling or landfill operations. There are some toxic aspects of API-driven data operations, but when measured and controlled, and made more observable, the dangerous aspects can be mitigated, and you might also find ways that other reuse and extraction that can also occur along the way.Â
If you think there is a link I should have listed here feel free to tweet it at me, or submit as a Github issue. Even though I do this full time, I'm still a one person show, and I miss quite a bit, and depend on my network to help me know what is going on.