Networking Cookbook
Everything you need to know about HTTP, JSON and REST API to get started.
HTTP
What is HTTP?
HTTP stands for Hypertext Transfer Protocol.
HTTP is the network protocol of the Web. It is both simple and powerful. Knowing HTTP enables you to write Web browsers, Web servers, automatic page downloaders, link-checkers, and other useful tools.
It's the network protocol used to deliver virtually all files and other data (collectively called resources) on the World Wide Web, whether they're HTML files, image files, query results, or anything else. Usually, HTTP takes place through TCP/IP sockets (and this tutorial ignores other possibilities).
A browser is an HTTP client because it sends requests to an HTTP server (Web server), which then sends responses back to the client. The standard (and default) port for HTTP servers to listen on is 80, though they can use any port.
What are "Resources"?
HTTP is used to transmit resources, not just files. A resource is some chunk of information that can be identified by a URL (it's the R in URL).
Structure
The format of the request and response messages are similar and English-oriented. Both kinds of messages consist of:
- an initial line,
- zero or more header lines,
- a blank line (i.e. a CRLF by itself), and
- an optional message body (e.g. a file, or query data, or query output).
<initial line, different for request vs. response>
Header1: value1
Header2: value2
Header3: value3
<optional message body goes here, like file contents or query data;
it can be many lines long, or even binary data $&*%@!^$@>Initial Request Line
The initial line is different for the request than for the response. A request line has three parts, separated by spaces: a method name, the local path of the requested resource, and the version of HTTP being used. A typical request line is:
GET /path/to/file/index.html HTTP/1.0
Notes:
- GET is the most common HTTP method; it says "give me this resource". Other methods include POST, DELETE, PUT and HEAD. Method names are always uppercase.
- The path is the part of the URL after the host name, also called the request URI (a URI is like a URL, but more general).
- The HTTP version always takes the form "HTTP/x.x", uppercase.
Initial Response Line (Status Line)
The initial response line, called the status line, also has three parts separated by spaces: the HTTP version, a response status code that gives the result of the request, and an English reason phrase describing the status code. Typical status lines are:
HTTP/1.0 200 OK
or
HTTP/1.0 404 Not Found
Notes:
- The HTTP version is in the same format as in the request line, "HTTP/x.x".
- The status code is meant to be computer-readable; the reason phrase is meant to be human-readable, and may vary.
- The status code is a three-digit integer, and the first digit identifies the general category of response:
- 1xx indicates an informational message only
- 2xx indicates success of some kind
- 3xx redirects the client to another URL
- 4xx indicates an error on the client's part
- 5xx indicates an error on the server's part
Most Common Status Codes
200 OK The request succeeded, and the resulting resource (e.g. file or script output) is returned in the message body.
201 Created The request is complete, and a new resource is created .
301 Moved Permanently302 Moved Temporarily
400 Bad Request The server did not understand the request.
401 Unauthorized The requested page needs a username and a password.
403 Forbidden Access is forbidden to the requested page.
404 Not Found The requested resource doesn't exist.
500 Server Error An unexpected server error. The most common cause is a server-side script that has bad syntax, fails, or otherwise can't run correctly.
503 Service Unavailable The request was not completed. The server is temporarily overloading or down.
A complete list of status codes is in the HTTP specification.
HTTP 1.1
Like many protocols, HTTP is constantly evolving. HTTP 1.1 has recently been defined, to address new needs and overcome shortcomings of HTTP 1.0. Generally speaking, it is a superset of HTTP 1.0. Improvements include:
- Faster response, by allowing multiple transactions to take place over a single persistent connection.
- Faster response and great bandwidth savings, by adding cache support.
- Faster response for dynamically-generated pages, by supporting chunked encoding, which allows a response to be sent before its total length is known.
- Efficient use of IP addresses, by allowing multiple domains to be served from a single IP address.
What’s JSON?
JSON is stands for “JavaScript Object Notation”, It’s used to store and exchange data as an alternative solution for XML. JSON is easier to parse, quicker to read and write. JSON sytnax is based on JavaScript Object Literals, But it’s a text format.
Keys and Values
The two primary parts that make up JSON are keys and values. Together they make a key/value pair.
- Key: A key is always a string enclosed in quotation marks.
- Value: A value can be a string, number, boolean expression, array, or object.
- Key/Value Pair: A key value pair follows a specific syntax, with the key followed by a colon followed by the value. Key/value pairs are comma separated.
Let's take one line from the JSON sample above and identify each part of the code.
"name" : "Pawan"This example is a key/value pair. The key is "name" and the value is "Pawan".
Types of Values
- Array: An associative array of values.
- Boolean: True or false.
- Number: An integer.
- Object: An associative array of key/value pairs.
- String: Several plain text characters which usually form a word.
Arrays
Almost every blog has categories and tags. In this example we've added a categories key, but the value might look unfamiliar. Since each post in a blog can have more than one category, an array of multiple strings is returned.
"person" : {
"name" : "Pawan",
"skills" : [ "Flutter", "React" ]
}Objects
An object is indicated by curly brackets. Everything inside of the curly brackets is part of the object. We already learned a value can be an object. So that means "person" and the corresponding object are a key/value pair.
"person" : {
"name" : "Pawan"
}REST API
There’s a high chance you came across the term “REST API” if you’ve thought about getting data from another source on the internet, such as Twitter or Github. But what is a REST API? What can it do for you? How do you use it?
What Is A REST API
Let’s say you’re trying to find videos about Flutter on Youtube. You open up Youtube, type “Flutter MTECHVIRAL” into a search field, hit enter, and you see a list of videos about Flutter. A REST API works in a similar way. You search for something, and you get a list of results back from the service you’re requesting from.
An API is an application programming interface. It is a set of rules that allow programs to talk to each other. The developer creates the API on the server and allows the client to talk to it.
REST determines how the API looks like. It stands for “Representational State Transfer”. It is a set of rules that developers follow when they create their API. One of these rules states that you should be able to get a piece of data (called a resource) when you link to a specific URL.
Each URL is called a request while the data sent back to you is called a response.
API Versions
Developers update their APIs from time to time. Sometimes, the API can change so much that the developer decides to upgrade their API to another version. If this happens, and your application breaks, it’s usually because you’ve written code for an older API, but your request points to the newer API.
You can request for a specific API version in two ways. Which way you choose depends on how the API is written.
These two ways are:
- Directly in the endpoint
- In a request header
Twitter, for example, uses the first method. At the time of writing, Twitter’s API is at version 1.1, which is evident through its endpoint:
https://api.twitter.com/1.1/account/settings.jsonGithub, on the other hand, uses the second method. At the time of writing, Github’s API is at version 3, and you can specify the version with an Accept header:
curl https://api.github.com -H Accept:application/vnd.github.v3+json
The Anatomy Of A Request
It’s important to know that a request is made up of four things:
- The endpoint
- The method
- The headers
- The data (or body)
1) The endpoint (or route) is the url you request for. It follows this structure:
root-endpoint/?
The root-endpoint is the starting point of the API you’re requesting from. The root-endpoint of Github’s API is https://api.github.com while the root-endpoint Twitter’s API is https://api.twitter.com.
The path determines the resource you’re requesting for. Think of it like an automatic answering machine that asks you to press 1 for a service, press 2 for another service, 3 for yet another service and so on.
You can access paths just like you can link to parts of a website. For example, to get a list of all posts tagged under “Flutter” on Codepur, you navigate to https://www.codepur.dev/tag/flutter/. https://www.codepur.dev/ is the root-endpoint and /tag/flutter is the path.
To understand what paths are available to you, you need to look through the API documentation. For example, let’s say you want to get a list of repositories by a certain user through Github’s API. The docs tells you to use the the following path to do so:
/users/:username/repos
Any colons (:) on a path denotes a variable. You should replace these values with actual values of when you send your request. In this case, you should replace :username with the actual username of the user you’re searching for. If I’m searching for my Github account, I’ll replace :username with iampawan.
The endpoint to get a list of my repos on Github is this:
https://api.github.com/users/iampawan/reposThe final part of an endpoint is query parameters. Technically, query parameters are not part of the REST architecture, but you’ll see lots of APIs use them. So, to help you completely understand how to read and use API’s we’re also going to talk about them. Query parameters give you the option to modify your request with key-value pairs. They always begin with a question mark (?). Each parameter pair is then separated with an ampersand (&), like this:
?query1=value1&query2=value2
When you try to get a list of a user’s repositories on Github, you add three possible parameters to your request to modify the results given to you:
If you’d like to get a list the repositories that I pushed to recently, you can set sortto push.
https://api.github.com/users/iampawan/repos?sort=pushedTesting Endpoints
You can send a request with any programming language. JavaScript users can use methods like the Fetch API and jQuery’s Ajax method; Flutter or Dart users can use HTTP package; Ruby users can use Ruby’s Net::HTTP class, Python users can use Python Requests; and so on.
You can also use tools like POSTMAN.
2) The Method
The method is the type of request you send to the server. You can choose from these five types below:
GETThis request is used to get a resource from a server.POSTThis request is used to create a new resource on a server.PUTThis request is used to update a resource on a server.PATCHThis request is used to update a resource on a server.DELETEThis request is used to delete a resource from a server.
This is the same thing we discussed in the HTTP Section
These methods provide meaning for the request you’re making. They are used to perform four possible actions: Create, Read, Update and Delete (CRUD).
3) The Headers
Headers are used to provide information to both the client and server. It can be used for many purposes, such as authentication and providing information about the body content. You can find a list of valid headers on MDN’s HTTP Headers Reference.
HTTP Headers are property-value pairs that are separated by a colon. The example below shows a header that tells the server to expect JSON content.
"Content-Type: application/json".
You can send HTTP headers with curl through the -H or --header option. To send the above header to Github’s API, you use this command:
curl -H "Content-Type: application/json" https://api.github.com4) The Data (Or “Body”)
The data (sometimes called “body” or “message”) contains information you want to be sent to the server. This option is only used with POST, PUT, PATCH or DELETErequests.
To send data through cURL, you can use the -d or --data option:
curl -X POST <URL> -d property1=value1
If you know how to spin up a server, you can make an API and test your own data. If you don’t know, but feel courageous enough to try, you can follow the link below to learn to create a server with Node, Express, and MongoDB
Pawan Kumar
If you don’t want to spin up your server, you can go to Requestbin.com (it’s free!) and click on the “create endpoint”. You’ll be given a URL that you can use to test requests, like https://en8brj58lmty9.x.pipedream.net/ shown in the picture below.
Authentication
You wouldn’t allow anyone to access your bank account without your permission, would you? On the same line of thought, developers put measures in place to ensure you perform actions only when you’re authorized to do. This prevents others from impersonating you.
Since POST, PUT, PATCH and DELETE requests alter the database, developers almost always put them behind an authentication wall. In some cases, a GETrequest also requires authentication (like when you access your bank account to check your current balance, for example).
On the web, there are two main ways to authenticate yourself:
- With a username and password (also called basic authentication)
- With a secret token
The secret token method includes oAuth, which lets you to authenticate yourself with social media networks like Github, Google, Twitter, Facebook, etc.
HTTP Status Codes And Error Messages
The final part is what to send back from a server to tell about the response by which a decision can be taken by the client. HTTP status codes let you tell the status of the response quickly. The range from 100+ to 500+.
Some of the messages, like “Requires authentication” and “Problems parsing JSON” are error messages.
We already discussed about the status codes in the HTTP section on the top.
Documentation
There is a great and quite widespread open-source framework called Swagger.io. It makes the life of developers easier in terms of documentation, description and even initial testing.
It also provides an opportunity to view the description of each request model. You can fill the fields with data, send it and get real results.
Demo
Conclusion
Unfortunately, not everyone understands how HTTP, JSON & RESTful API works. There are many cases when developers who deal with REST design architecture incorrectly. Make sure that your work with this style of writing API meets standards and that it is effective. Its proper use will save your time and efforts.
We hope that this cookbook will help you to understand the basics.