Home

Published

- 11 min read

Building Mr. Clip, my personal (but advanced) URL shortening app

Geolocation IP URL MySql Tomcat Spring boot
img of Building Mr. Clip, my personal (but advanced) URL shortening app

Introduction

Long links aren’t practical, especially if they’re shown on a presentation slide or on a brochure with an intent of being copied. Manually typing these links is tedious, for this reason URL shortening apps are popular. You’ve probably used some of them already or at least heard of them.

The basic premise of any URL shortener is creating a short URL which acts as an alias to another longer one, going to short link would redirect you -sometimes seamlessly sometimes with ads- to the intended site with the long URL.

Example

   mrclip.me/abcdef → https://example.org/path/to/resource/section/subsection/item/details

So how do we go about designing such app?

Basic implementation

A basic implementation of this app would go as the following:

Creating a short URL

1. User submits the long URL.

2. Application generates a short alias.

3. System stores the long URL and short alias pair in the database.

4. Application returns the short URL to the user.

Going to the intended site

  1. User goes the alias URL.
  2. The app retrieves the long URL and short alias pair in the database.
  3. If the pair doesn’t exist then a Not Found message is returned
  4. If it’s found then the user is redirected to the intended site (we’ll talk more about how this is done below)

Done, this covers the basic non-technical implementation which can be done in any framework or language.

For my implementation I decided to use Spring boot for the backend and MySQL as the database of choice.

Technical Implementation

Backend (Spring Boot & MySQL)

Two endpoints are needed

Shorten a URL

This endpoint receives a URL string via a post request and passes it to the responsible service

   @PostMapping("/create")
public ResponseEntity<ShortURLDTO> createShortURL(
        @RequestParam(name = "url") @URL @NotBlank String url) {
    var createdShortURLDTO = shortURLService.createShortURL(url).mapToDTO();
    return ResponseEntity.status(HttpStatus.CREATED).body(createdShortURLDTO);
}

The service does the following:

  • Generate an alias defined as code that’s 4 to 7 characters long inclusive
  • Stored the code & the long URL pair in the database
  • Returns the full short URL. E.g., https://mrclip.me/abcd
Code generation

The way the code is generated should take into account the readability and how easy it is to write the URL, we can only use certain characters and symbols in the URL, specifically the Unreserved characters per the RFC 3986 Standard; Section 2.3. These characters are the best to work with because they do not need to be encoded and thus offer the best readability and write-ability (is that a word?)

Code generation logic

   private String generateCode() {
    final int MAX_CODE_LENGTH = 7;
    final int MIN_CODE_LENGTH = 4;
    final int CODE_LENGTH = secureRandom.nextInt(MIN_CODE_LENGTH, MAX_CODE_LENGTH + 1);
//        Based on the RFC 3986 standard section 2.3
    final String UNRESERVED_CHARS = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz-._~";
    StringBuilder sb = new StringBuilder(CODE_LENGTH);
    for (int i = 0; i < CODE_LENGTH; i++) {
        int index = secureRandom.nextInt(UNRESERVED_CHARS.length());
        sb.append(UNRESERVED_CHARS.charAt(index));
    }
    return sb.toString();
}

The number of unreserved characters is 66, and with a code of length 7, we get 66^7 = 5,455,160,701,056, which is five trillion+ possible strings. But this is only if we use 7; we could use a length of 4 up to and including 7, and get the ability to generate even shorter URLs (which maybe only premium members could have), and a number of possible strings greater than five trillion.

You can also notice that secureRandom was used, this is only so we have random character generation without implementing our own. I’d say the security part isn’t very crucial in this piece of code because I wouldn’t worry about timing attacks.

Code validation

Rest of service logic

   public ShortURL createShortURL(String url) throws ShortURLGenerationFailureException {
    final int MAX_TRIES = 5;
    int numberOfTries = 1;
    while (numberOfTries <= MAX_TRIES) {
        var code = generateCode();
        if (this.shortURLRepository.existsByCode(code)) {
            numberOfTries++;
            continue;
        }
        var shortURL = ShortURL
                .builder()
                .code(code)
                .url(url)
                .build();
        this.shortURLRepository.save(shortURL);
        return shortURL;
    }
    throw new ShortURLGenerationFailureException();
}

In the above service logic is where the database entry for the code & long URL pair is saved and returned to the user. You can notice that I check if there’s already an entry in the database that has the code I’ve just generated, if that’s the case then I try again with a maximum trial count of 5. Five was chosen as an arbitrary number, if I failed to generate a unique code for five consecutive times then something must be wrong and an exception is thrown.

Exception class

   public class ShortURLGenerationFailureException extends RuntimeException {
  public ShortURLGenerationFailureException() {
    super("Failed to generate short url. Please try again later.");
  }
}

Which is handled by a ControllerAdvice that returns an HTTP status 503 (Service Unavailable)

   @ExceptionHandler(ShortURLGenerationFailureException.class)
public ResponseEntity<?> handleShortURLGenerationFailureException(ShortURLGenerationFailureException e) {
    return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE).body(e.getMessage());
}

This concludes the URL Shortening section, now how is the code retrieved by a user?

Going to the intended site

Through a GET endpoint, the user can send a code as a path variable which maps to a site, if a site is found then an HTTP status 308 (Permanent Redirect) would be returned with the header Location being set to the long URL. When browsers or most web clients see HTTP status 308, they automatically redirect the user to whatever URL set as the value of the Location header.

   @GetMapping("/{code}")
public ResponseEntity<Void> redirect(@PathVariable String code, HttpServletRequest request) {
    var shortURL = shortURLService.getShortURL(code);
    if (shortURL.isEmpty())
        throw new CodeNotFoundException(code);
    // Hidden analytics logic which will be discussed later
    var url = shortURL.get().getUrl();
    var headers = new HttpHeaders();
    headers.set(HttpHeaders.LOCATION, url);

    return ResponseEntity.status(HttpStatus.PERMANENT_REDIRECT).headers(headers).build();
}

This concludes the technical implementation of the basic premise of a URL shortening app, but we can do more.

Analytics

Let’s offer the ability for users to see analytics for the short links they create.

The analytics we’re interested are

  • User’s device
  • IP
  • Country
  • Browser
  • Device
  • Time they clicked the link

We can add more but for now this suffices.

   // Schema
public class AnalyticsRecord {
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    @Id
    private Long id;

    @NotNull
    @ManyToOne(optional = false)
    @JoinColumn(name = "shorturl_id", nullable = false)
    private ShortURL shortURL;
    @NotNull
    private Instant accessTime;
    @NotNull
    private String ip;
    private String country;
    private String OS;
    private String device;
    private String browser;

    @PrePersist
    private void onCreate() {
        this.accessTime = Instant.now();
    }
}

Getting the time is trivial, use JPA to create an Instant instance before the record is saved and done. As for the others some work is needed.

Getting the User Agent

User agent contains information about the client which they voluntarily offer, such as the web client used (browsers are web clients), the OS it’s running on and its version, the device, the resolution and other information. However since it’s voluntarily information some web clients provide empty user agent. So in short you shouldn’t expect a user agent to always be present.

To parse the user agent provided I use the uap-parse library, it takes the user agent provided in the headers and a client object with all available information.

The GET redirection endpoint

   @GetMapping("/{code}")
public ResponseEntity<Void> redirect(@PathVariable String code, HttpServletRequest request) {
    var shortURL = shortURLService.getShortURL(code);
    if (shortURL.isEmpty())
        throw new CodeNotFoundException(code);
    // Extract user agent from header
    var userAgent = request.getHeader(HttpHeaders.USER_AGENT);
    // Conctruct client object
    var clientUserAgent = uaParser.parse(userAgent);
    // pass it to service
    this.analyticsService.recordClick(shortURL.get(), request.getRemoteAddr(), clientUserAgent, "JO");

    var url = shortURL.get().getUrl();
    var headers = new HttpHeaders();
    headers.set(HttpHeaders.LOCATION, url);

    return ResponseEntity.status(HttpStatus.PERMANENT_REDIRECT).headers(headers).build();
}

The service

   public void recordClick(ShortURL shortURL, String clientIP,
                                                // We haven't discussed how to determine the country yet
                        Client clientUserAgent, String countryCode) {
    var click = AnalyticsRecord.builder()
            .shortURL(shortURL)
            .country(countryCode)
            .ip(clientIP)
            .browser(clientUserAgent.userAgent.family)
            .OS(clientUserAgent.os.family)
            .device(clientUserAgent.device.family)
            .build();
    analyticsRepository.save(click);
}

That’s it regarding the user agent, the uap-parse takes care of all the work for us and we only need to supply it with the user agent header. Now we go to the harder part which is determining the IP and thus the country of the user.

The IP and Country

You don’t actually need the IP to find the country, it’s one of the ways you could do that, but not the only one. You can try determining the location from:

  • The user’s timezone. However since multiple countries can have the same timezone this is not accurate.
  • Asking the user for their location explicitly via the browser’s navigator API. Obviously users wouldn’t be too keen on giving you their location, plus UX would suffer.
  • Inferring from their used language again using the navigator API. Still inaccurate because users might choose a language setting unrelated to their location, such as British English but they’re in Jordan.

This leaves us with the IP which has the highest chance (still not guaranteed) of finding the location of the user.

IP

To determine the IP where the request originates from I added a HttpServletRequest argument to the GET controller. This exposes information about the request such as headers as you’ve seen above, cookies and remote address which is the IP.

So if we do a simple getRemoteAddr we’re done right? Wrong, unfortunately it’s never that simple, and the reason for this here are proxies.

A proxy server is an intermediate program or computer used when navigating through different networks of the Internet. They facilitate access to content on the World Wide Web. A proxy intercepts requests and serves back responses; it may forward the requests, or not (for example in the case of a cache), and it may modify it (for example changing its headers, at the boundary between two networks).

MDN

I use Cloudflare to proxy requests from clients to my server (VPS). Cloudflare offers domain management and DDOS protection and this is why I use it.

When a user sends a request to shorten a URL it first goes to the Cloudflare servers then to my server, so simply calling getRemoteAddr on the request would give back the IP of the Cloudflare proxy server that the client request passed through.

Inconvenient, but there’s away around this. By convention, proxy services and programs such as Cloudflare and Nginx insert the actual IP of the user into the header X-Forwarded-For this header contains the actual IP of the user, not the proxy server.

Notice that I said “By convention” this is because proxy servers don’t have to give you the actual IP and in practice many proxies don’t include it in the X-Forwarded-For header for security purposes. This happens mostly in the case where the users set their own proxy servers to protect themselves, but in my case Cloudflare is configured to provide the actual IP in the headers.

In summary

   //  Fails in case of proxy
var remoteAdd = request.getRemoteAddr();
//  Works in case of proxy, but fails if there's no proxy
var forwardedFor = request.getHeader("X-Forwarded-For");
                                                                            // could be a list so watch out
String ip = (forwardedFor == null || forwardedFor.isEmpty()) ? remoteAdd : forwardedFor;

Because I use Tomcat as my application’s web server, I can set the following environment variable server.tomcat.remote-ip-header=x-forwarded-for this replaces the apparent client remote IP address and hostname for the request with the IP address list presented by a proxy or a load balancer and so I can just use getRemoteAddr and I’ll be confident I’ll always get the actual IP of the user.

Country

To determine the country from the IP we’ll use the GeoLite2 database which is a free to use database that maps IP addresses to geolocation. I’ll get mine from this github repo and I’ll only use the countries database since I’m not interested in any finer location such as the city.

Loading the database

   @Configuration
public class GeoIPConfig {
    @Bean
    public DatabaseReader GeoIP() throws IOException {
        File database = new ClassPathResource("geoip/GeoLite2-Country.mmdb").getFile();
        return new DatabaseReader.Builder(database).build();
    }
}

And to use it I’ll just autowire this bean and call it inside the relevant analytics service

   public void recordClick(ShortURL shortURL, String clientIP,
                        Client clientUserAgent) {
    String countryCode = null;
    try {
        countryCode = geoIP.country(InetAddress.getByName(clientIP)).getCountry().getIsoCode();
    } catch (IOException | GeoIp2Exception e) {
        // you might be interested in logging the exception which occures if the mapping fails
    }

    var click = AnalyticsRecord.builder()
            .shortURL(shortURL)
            .country(countryCode)
            .ip(clientIP)
            // Check forn nulls in case user agent header was not provided
            .browser(clientUserAgent.userAgent != null ? clientUserAgent.userAgent.family : null)
            .OS(clientUserAgent.os != null ? clientUserAgent.os.family : null)
            .device(clientUserAgent.device != null ? clientUserAgent.device.family : null)
            .build();
    analyticsRepository.save(click);
}

and voila! we’re done. Now we have analytics ready for this our URL shortening app.

Conclusion

We learned how to build a URL shortening app with Spring Boot and MySQL. We had to deal with Tomcat, IPs, proxies and Geolocation. It’s interesting how such a simple concept can require so many steps, but that’s the nature of programming.

If you want to try Mr. Clip for yourself please go here: https://mrclip.me

The backend source code is hosted here: https://github.com/Asomeones222/shortly-spring.git

I hope you enjoyed this article, if you have any questions feel free to visit my socials above and send me a message!