Skip to content
This repository has been archived by the owner on May 1, 2024. It is now read-only.

The goal of this project is to get a list of games and their scores from a website. The application must parse the website HTML content, get the necessary information, save the game score data in a database and expose them through a REST API.

Notifications You must be signed in to change notification settings

ivangfr/springboot-jsoup-html-parser

Repository files navigation

springboot-jsoup-html-parser

The goal of this project is to get a list of games and their scores from a website. The application must parse the website HTML content, get the necessary information, save the data in a database and expose them through a REST API.

Note: In kubernetes-minikube-environment repository, it's shown how to deploy this project in Kubernetes (Minikube)

Proof-of-Concepts & Articles

On ivangfr.github.io, I have compiled my Proof-of-Concepts (PoCs) and articles. You can easily search for the technology you are interested in by using the filter. Who knows, perhaps I have already implemented a PoC or written an article about what you are looking for.

Project Diagram

project-diagram

Applications

  • game-score-api

    Spring Boot Java Web application that exposes a REST API from where clients can retrieve the game score data stored in MongoDB database.

    Endpoints Description
    GET /api/games[?title] returns all game scores or filtered by title with pagination
    GET /api/games/{id} returns a specific game score filtered by id
  • game-score-collector

    Spring Boot Java application responsible for calling the game score website, parse the HTML content (using jsoup) and save the data in MongoDB database. It will be configured to run from time to time in order to keep the application updated about the information the website provides.

Prerequisites

Start Environment

Open a terminal and, inside springboot-jsoup-html-parser root folder, the command below

docker compose up -d

Running Applications with Maven

  • In a terminal, make sure you are inside springboot-jsoup-html-parser root folder

  • Execute the following command to run game-score-collector

    ./mvnw clean spring-boot:run --projects game-score-collector
    

    game-score-collector is a Java application that does its job and terminates. Ideally, it will be executed as a cronjob, scheduled to run during specific time intervals.

  • Execute the command below to run game-score-api

    ./mvnw clean spring-boot:run --projects game-score-api
    

Running Applications as Docker containers

Build Docker Images

  • In a terminal, make sure you are in springboot-jsoup-html-parser root folder

  • Run the following script to build the Docker images

    • JVM
      ./docker-build.sh
      
    • Native
      ./docker-build.sh native
      

Environment Variables

  • game-score-api

    Environment Variable Description
    MONGODB_HOST Specify host of the Mongo database to use (default localhost)
    MONGODB_PORT Specify port of the Mongo database to use (default 27017)
  • game-score-collector

    Environment Variable Description
    MONGODB_HOST Specify host of the Mongo database to use (default localhost)
    MONGODB_PORT Specify port of the Mongo database to use (default 27017)

Start Docker Containers

  • In a terminal, execute the command below to run game-score-collector Docker container

    docker run --rm --name game-score-collector \
      -e MONGODB_HOST=mongodb \
      --network=springboot-jsoup-html-parser_default \
      ivanfranchin/game-score-collector:1.0.0
    
  • Then, execute the following command to run game-score-api Docker container

    docker run --rm --name game-score-api -p 8080:8080 \
      -e MONGODB_HOST=mongodb \
      --network=springboot-jsoup-html-parser_default \
      ivanfranchin/game-score-api:1.0.0
    

Testing

  • In a terminal, the following curl command returns the game score results with pagination: page 0, size 10, sorted descending by score field.

    curl -i "http://localhost:8080/api/games?page=0&size=10&sort=score%2Cdesc"
    

    You can get a specific game score by running the following curl command

    curl -i http://localhost:8080/api/games/325927
    
  • You can access game-score-api Swagger at http://localhost:8080

Useful Commands

  • MongoDB

    List all game scores

    docker exec -it mongodb mongosh gamescoredb
    db.gamescores.find()
    db.gamescores.getIndexes()
    

    Type exit to get out of MongoDB shell

Shutdown

  • To stop game-score-api, go to the terminal where it is running and press Ctrl+C
  • To stop and remove docker compose containers, network and volumes, go to a terminal and, inside springboot-jsoup-html-parser root folder, run the following command
    docker compose down -v
    

Running Tests

Both game-score-api and game-score-collector have a set of test cases. In order to run them

  • In a terminal, make sure you are inside springboot-jsoup-html-parser root folder
  • Execute the following command
    ./mvnw clean test --projects game-score-collector,game-score-api
    

Cleanup

To remove the Docker images created by this project, go to a terminal and, inside springboot-jsoup-html-parser root folder, run the following script

./remove-docker-images.sh

About

The goal of this project is to get a list of games and their scores from a website. The application must parse the website HTML content, get the necessary information, save the game score data in a database and expose them through a REST API.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Sponsor this project

 

Packages

No packages published