[{"content":"Hello, I\u0026rsquo;m Angelos, a Software Engineer specializing in backend and frontend development.\nFavourite languages/tools for software development?\nI work mainly with Go and Python on the backend, and React/TypeScript on the frontend. This site is built with Hugo and the Congo theme.\nOther interests?\nI enjoy listening to heavy metal, playing electric guitar, reading science-fiction and fantasy, PC gaming (especially RPGs), and staying active through running and exercise.\nDownload my CV\nCertifications # ","date":null,"permalink":"https://www.angelospanag.me/","section":"Angelos Panagiotopoulos","summary":"","title":"Angelos Panagiotopoulos"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/aws/","section":"Tags","summary":"","title":"Aws"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/cognito/","section":"Tags","summary":"","title":"Cognito"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/fastapi/","section":"Tags","summary":"","title":"Fastapi"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/jwt/","section":"Tags","summary":"","title":"Jwt"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/posts/","section":"Posts","summary":"","title":"Posts"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/python/","section":"Tags","summary":"","title":"Python"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/","section":"Tags","summary":"","title":"Tags"},{"content":" NOTE: This post is intentionally structured similarly to Verifying a JSON Web Token from Amazon Cognito in Go and Gin, but showcasing the same methodology using Python related technologies.\nIntro Retrieving a public JWKS Verifying an incoming JWT in a FastAPI Dependency 1. Defining a FastAPI Dependency that performs Cognito authentication 2. Retrieving the JWT from an incoming request 3. Verify the JWT signature, signing algorithm, issuer (iss) and existence of expiry time (exp) 4. Check the token_use claim 5. Verify the audience (aud)/client ID (client_id) claim Complete middleware snippet Using the Dependencies in a FastAPI endpoint Full implementation Intro #One popular option for integrating Amazon Cognito authentication/authorization with a backend, requires the decoding and verifying of JSON Web Tokens (JWT) for server-side processing. AWS details the steps required to validate an incoming JWT produced by Amazon Cognito in its official documentation, and offers an example using a dedicated JavaScript library.\nWe can implement the above steps in Python and FastAPI, using PyJWT.\nRetrieving a public JWKS #The JWKS URI contains public information about the private key that signed your user\u0026rsquo;s token. As soon as a Cognito User Pool is created, it will publish its JWKS in a public URI. It can be composed as followed https://cognito-idp.\u0026lt;region\u0026gt;.amazonaws.com/\u0026lt;userPoolId\u0026gt;/.well-known/jwks.json where region is the AWS region of your User Pool and userPoolId the ID of the User Pool.\nThe PyJWT Python library, has a convenient method jwt.PyJWKClient for retrieving that key using an HTTP request and parsing it, all in a single statement:\nimport jwt from fastapi_cognito_jwt.config import get_settings jwks_client = jwt.PyJWKClient( f\u0026#34;https://cognito-idp.{get_settings().aws_default_region}.amazonaws.com/{get_settings().cognito_user_pool_id}/.well-known/jwks.json\u0026#34; ) Verifying an incoming JWT in a FastAPI Dependency #1. Defining a FastAPI Dependency that performs Cognito authentication #FastAPI offers an elegant dependency injection mechanism in the form of Dependencies. It can also be leveraged to enforce authentication/authorization, and it is ideal for protecting endpoints using JWTs.\nEven better, a FastAPI Dependency can take the form of a callable instance of a class, which can be instantiated with values of our choosing in its __init__ method and where our JWT verification logic can be implemented inside its __call__ method.\nFollowing this convention, we can start implementing a dependency called CognitoJWTAuthorizer in a file dependencies.py, with the following function signature, passing the token use (required_token_use which can be access or id depending on the type of Cognito token we accept), AWS Region (aws_default_region), the Cognito User Pool ID (cognito_user_pool_id), the Cognito App Client ID that we\u0026rsquo;ve created (cognito_app_client_id) and the instantiated JWKS client containing the JWKS key (jwks_client).\nThe __call__ method can take as an argument a FastAPI header authorization: Annotated[str | None, Header()], where its variable name authorization is automatically mapped to the actual Authorization header of an incoming HTTP request.\nfrom enum import Enum from typing import Annotated import jwt from fastapi import Header, HTTPException from starlette.status import HTTP_401_UNAUTHORIZED from python_fastapi_cognito_jwt_verification.config import get_settings jwks_client = jwt.PyJWKClient( f\u0026#34;https://cognito-idp.{get_settings().aws_default_region}.amazonaws.com/{get_settings().cognito_user_pool_id}/.well-known/jwks.json\u0026#34; ) class CognitoTokenUse(Enum): ID = \u0026#34;id\u0026#34; ACCESS = \u0026#34;access\u0026#34; class CognitoJWTAuthorizer: def __init__( self, required_token_use: CognitoTokenUse, aws_default_region: str, cognito_user_pool_id: str, cognito_app_client_id: str, jwks_client: jwt.PyJWKClient, ) -\u0026gt; None: self.required_token_use = required_token_use self.aws_default_region = aws_default_region self.cognito_user_pool_id = cognito_user_pool_id self.cognito_app_client_id = cognito_app_client_id self.jwks_client = jwks_client def __call__(self, authorization: Annotated[str | None, Header()] = None): 2. Retrieving the JWT from an incoming request #By convention, an incoming request contains a JWT in its Authorization header using a Bearer token.\nThe Authorization header including the Bearer token has the format:\nAuthorization: Bearer abcdefg Using the variable authorization that we passed to the __call__ method we can retrieve the raw value of the JWT by splitting the string containing it.\nif not authorization: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) split_authorization_tokens: list[str] = authorization.split(\u0026#34;Bearer \u0026#34;) if len(split_authorization_tokens) \u0026lt; 2: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) token: str = split_authorization_tokens[1] 3. Verify the JWT signature, signing algorithm, issuer (iss) and existence of expiry time (exp) #Now, using PyJWT we can perform some first rudimentary checks against the JWT using some convenient methods offered by the library, also shown in its official documentation.\nFirst, we can retrieve the signing key of the JWT:\ntry: signing_key: jwt.PyJWK = self.jwks_client.get_signing_key_from_jwt(token) except jwt.exceptions.InvalidTokenError as e: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) from e Then we can attempt to perform the majority of the required checks against the JWT:\nThe signature of the JWT is verified using the jwt.decode method by providing the retrieved signing key, in the form of a siging_key.key argument, along with some extra checks we define below.\nOne of the most important verifications is defining the specific valid algorithm methods that the parser will use, and confirming that the incoming token is using those exclusively. In the case of Amazon Cognito the asymmetric algorithm RS256 is used to sign the key. This can be enforced using the algorithms argument and passing a list of algorithms, in our case just [\u0026quot;RS256\u0026quot;]. Leaving this option without a value leaves us open to algorithm confusion attacks.\nThe token should not be expired. The jwt.decode method can automate this check for us, by passing verify_exp: True as a key-pair in the options argument.\nThe issuer claim (iss) should match our User Pool. For example, a User Pool created in the us-east-1 region will have the following iss value: https://cognito-idp.us-east-1.amazonaws.com/\u0026lt;userpoolID\u0026gt;. We can verify this using the issuer argument and also pass verify_iss: True as a key-pair in the options argument, just to be extra explicit about this check.\nWe can also require the existence of specific claims in the token, by passing a list of them in the require attribute of the options argument. Our verification process needs the claims [\u0026quot;token_use\u0026quot;, \u0026quot;exp\u0026quot;, \u0026quot;iss\u0026quot;, \u0026quot;sub\u0026quot;].\nVerification of the audience claim (aud) is taken care later when we examine if the token is a Cognito \u0026lsquo;id\u0026rsquo; or \u0026lsquo;access\u0026rsquo; token. That is why for now we pass verify_aud: False as a key-pair in the options argument.\nThe above checks can be succinctly defined in one statement, facilitated by PyJWT, as followed:\ntry: claims = jwt.decode( token, signing_key.key, algorithms=[\u0026#34;RS256\u0026#34;], issuer=f\u0026#34;https://cognito-idp.{self.aws_default_region}.amazonaws.com/{self.cognito_user_pool_id}\u0026#34;, options={ \u0026#34;verify_aud\u0026#34;: False, \u0026#34;verify_signature\u0026#34;: True, \u0026#34;verify_exp\u0026#34;: True, \u0026#34;verify_iss\u0026#34;: True, \u0026#34;require\u0026#34;: [\u0026#34;token_use\u0026#34;, \u0026#34;exp\u0026#34;, \u0026#34;iss\u0026#34;, \u0026#34;sub\u0026#34;], }, ) except jwt.exceptions.ExpiredSignatureError as e: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) from e except jwt.exceptions.InvalidTokenError as e: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) from e 4. Check the token_use claim #Cognito can send ID or access tokens, each with a different set of attributes. Depending on the nature of the endpoint we want to protect we can choose to accept specific types.\nID tokens contain claims about the identity of the authenticated user, such as name, email, and phone_number.\nAccess tokens contain claims about the authorization of the authenticated user such as a list of the user\u0026rsquo;s groups, and a list of scopes.\nif self.required_token_use.value != claims[\u0026#34;token_use\u0026#34;]: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) 5. Verify the audience (aud)/client ID (client_id) claim #Depending on the type of token (access or ID), we can check respectively the aud or the client_id claims and that they should match the Cognito App Client ID created in the Cognito User Pool.\nFor each case, we can check the existence of aud the client_id custom claims in claims, the same way we would check for values in a Python dictionary, and compare their value against the Cognito App Client ID.\nif self.required_token_use == CognitoTokenUse.ID: if \u0026#34;aud\u0026#34; not in claims: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) if claims[\u0026#34;aud\u0026#34;] != self.cognito_app_client_id: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) elif self.required_token_use == CognitoTokenUse.ACCESS: if \u0026#34;client_id\u0026#34; not in claims: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) if claims[\u0026#34;client_id\u0026#34;] != self.cognito_app_client_id: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) else: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) Complete middleware snippet #The complete snippet using all the above statements is as followed:\nfrom enum import Enum from typing import Annotated import jwt from fastapi import Header, HTTPException from starlette.status import HTTP_401_UNAUTHORIZED from python_fastapi_cognito_jwt_verification.config import get_settings jwks_client = jwt.PyJWKClient( f\u0026#34;https://cognito-idp.{get_settings().aws_default_region}.amazonaws.com/{get_settings().cognito_user_pool_id}/.well-known/jwks.json\u0026#34; ) class CognitoTokenUse(Enum): ID = \u0026#34;id\u0026#34; ACCESS = \u0026#34;access\u0026#34; class CognitoJWTAuthorizer: def __init__( self, required_token_use: CognitoTokenUse, aws_default_region: str, cognito_user_pool_id: str, cognito_app_client_id: str, jwks_client: jwt.PyJWKClient, ) -\u0026gt; None: \u0026#34;\u0026#34;\u0026#34;Verify an incoming JWT using official AWS guidelines. https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html#amazon-cognito-user-pools-using-tokens-manually-inspect Args: required_token_use (CognitoTokenUse): Required Cognito token type aws_default_region (str): AWS region cognito_user_pool_id (str): Cognito User Pool ID cognito_app_client_id (str): Cognito App Pool ID jwks_client (jwt.PyJWKClient): An instance of a JWKS client that has retrieved the public Cognito JWKS Raises: fastapi.HTTPException: Raised if a verification check of the incoming JWT fails \u0026#34;\u0026#34;\u0026#34; self.required_token_use = required_token_use self.aws_default_region = aws_default_region self.cognito_user_pool_id = cognito_user_pool_id self.cognito_app_client_id = cognito_app_client_id self.jwks_client = jwks_client def __call__(self, authorization: Annotated[str | None, Header()] = None): \u0026#34;\u0026#34;\u0026#34;Verify an embedded Cognito JWT token in the \u0026#39;Authorization\u0026#39; header. Args: authorization (str | None): \u0026#39;Authorization\u0026#39; header of a FastAPI endpoint \u0026#34;\u0026#34;\u0026#34; if not authorization: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) split_authorization_tokens: list[str] = authorization.split(\u0026#34;Bearer \u0026#34;) if len(split_authorization_tokens) \u0026lt; 2: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) token: str = split_authorization_tokens[1] # Attempt to retrieve the signature of the incoming JWT try: signing_key: jwt.PyJWK = self.jwks_client.get_signing_key_from_jwt(token) except jwt.exceptions.InvalidTokenError as e: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) from e try: \u0026#34;\u0026#34;\u0026#34; * Verify the signature of the JWT * Verify that the algorithm used is RS256 * Verification of audience \u0026#39;aud\u0026#39; is taken care later when we examine if the token is \u0026#39;id\u0026#39; or \u0026#39;access\u0026#39; * Verify that the token hasn\u0026#39;t expired. Decode the token and compare the \u0026#39;exp\u0026#39; claim to the current time. * The issuer (iss) claim should match your user pool. For example, a user pool created in the eu-west-2 region will have the following iss value: https://cognito-idp.us-east-1.amazonaws.com/\u0026lt;userpoolID\u0026gt;. * Require the existence of claims: \u0026#39;token_use\u0026#39;, \u0026#39;exp\u0026#39;, \u0026#39;iss\u0026#39;, \u0026#39;sub\u0026#39; \u0026#34;\u0026#34;\u0026#34; claims = jwt.decode( token, signing_key.key, algorithms=[\u0026#34;RS256\u0026#34;], issuer=f\u0026#34;https://cognito-idp.{self.aws_default_region}.amazonaws.com/{self.cognito_user_pool_id}\u0026#34;, options={ \u0026#34;verify_aud\u0026#34;: False, \u0026#34;verify_signature\u0026#34;: True, \u0026#34;verify_exp\u0026#34;: True, \u0026#34;verify_iss\u0026#34;: True, \u0026#34;require\u0026#34;: [\u0026#34;token_use\u0026#34;, \u0026#34;exp\u0026#34;, \u0026#34;iss\u0026#34;, \u0026#34;sub\u0026#34;], }, ) except jwt.exceptions.ExpiredSignatureError as e: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) from e except jwt.exceptions.InvalidTokenError as e: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) from e \u0026#34;\u0026#34;\u0026#34; Check the token_use claim * If you are only accepting the access token in your web API operations, its value must be access. * If you are only using the ID token, its value must be id. * If you are using both ID and access tokens, the token_use claim must be either id or access. \u0026#34;\u0026#34;\u0026#34; if self.required_token_use.value != claims[\u0026#34;token_use\u0026#34;]: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) \u0026#34;\u0026#34;\u0026#34; The \u0026#34;aud\u0026#34; claim in an ID token and the \u0026#34;client_id\u0026#34; claim in an access token should match the app client ID that was created in the Amazon Cognito user pool. \u0026#34;\u0026#34;\u0026#34; if self.required_token_use == CognitoTokenUse.ID: if \u0026#34;aud\u0026#34; not in claims: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) if claims[\u0026#34;aud\u0026#34;] != self.cognito_app_client_id: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) elif self.required_token_use == CognitoTokenUse.ACCESS: if \u0026#34;client_id\u0026#34; not in claims: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) if claims[\u0026#34;client_id\u0026#34;] != self.cognito_app_client_id: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) else: raise HTTPException( status_code=HTTP_401_UNAUTHORIZED, detail=\u0026#34;Unauthorized\u0026#34; ) cognito_jwt_authorizer_access_token = CognitoJWTAuthorizer( CognitoTokenUse.ACCESS, get_settings().aws_default_region, get_settings().cognito_user_pool_id, get_settings().cognito_app_client_id, jwks_client, ) cognito_jwt_authorizer_id_token = CognitoJWTAuthorizer( CognitoTokenUse.ID, get_settings().aws_default_region, get_settings().cognito_user_pool_id, get_settings().cognito_app_client_id, jwks_client, ) Using the Dependencies in a FastAPI endpoint #Putting it all together, we can create FastAPI endpoints that are protected using the above Dependencies.\nThe below code can be used in a main.py file and shows how to define two protected FastAPI endpoints, one requiring a Cognito ID token (/protected-with-id-token) and one requiring a Cognito Access Token (/protected-with-access-token).\nfrom fastapi import Depends, FastAPI from python_fastapi_cognito_jwt_verification.dependencies import ( cognito_jwt_authorizer_access_token, cognito_jwt_authorizer_id_token, ) app = FastAPI() @app.get(\u0026#34;/protected-with-access-token\u0026#34;, dependencies=[Depends(cognito_jwt_authorizer_access_token)]) def protected_with_access_token(): return {\u0026#34;Hello\u0026#34;: \u0026#34;World\u0026#34;} @app.get(\u0026#34;/protected-with-id-token\u0026#34;, dependencies=[Depends(cognito_jwt_authorizer_id_token)]) def protected_with_id_token(): return {\u0026#34;Hello\u0026#34;: \u0026#34;World\u0026#34;} Full implementation #You can find the fully working implementation in a GitHub repository.\n","date":"4 December 2023","permalink":"https://www.angelospanag.me/posts/verifying-a-json-web-token-from-cognito-in-python-and-fastapi/","section":"Posts","summary":"How to secure a Python backend using Amazon Cognito","title":"Verifying a JSON Web Token from Amazon Cognito in Python and FastAPI"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/gin/","section":"Tags","summary":"","title":"Gin"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/go/","section":"Tags","summary":"","title":"Go"},{"content":" NOTE: This post is intentionally structured similarly to Verifying a JSON Web Token from Amazon Cognito in Python and FastAPI, but showcasing the same methodology using Go related technologies.\nIntro Retrieving a public JWKS Verifying an incoming JWT in a Gin middleware 1. Defining a Gin authentication middleware 2. Retrieving the JWT from an incoming request 3. Verify the JWT signature, signing algorithm, issuer (iss) and existence of expiry time (exp) 4. Parse JWT claims 5. Compare the exp claim to the current time 6. Check the token_use claim 7. Verify the existence of the subject (sub) claim 8. Verify the audience (aud)/client ID (client_id) claim 9. Optionally retrieve any Cognito user groups that the user belongs to 10. Finally, proceed to the actual request Complete middleware snippet Using the middleware in a Gin endpoint Full implementation Intro #One popular option for integrating Amazon Cognito authentication/authorization with a backend, requires the decoding and verifying of JSON Web Tokens (JWT) for server-side processing. AWS details the steps required to validate an incoming JWT produced by Amazon Cognito in its official documentation, and offers an example using a dedicated JavaScript library.\nWe can implement the above steps in Go and Gin web framework, using a recommended combination of two modern and well-maintained Go libraries, taking advantage of the excellent interoperability between them.\nkeyfunc for consuming a JWKS and parsing it in an easily readable structure in Go jwt.Keyfunc.\ngolang-jwt for parsing and verifying JSON web tokens\nRetrieving a public JWKS #The JWKS URI contains public information about the private key that signed your user\u0026rsquo;s token. As soon as a Cognito User Pool is created, it will publish its JWKS in a public URI. It can be composed as followed https://cognito-idp.\u0026lt;region\u0026gt;.amazonaws.com/\u0026lt;userPoolId\u0026gt;/.well-known/jwks.json where region is the AWS region of your User Pool and userPoolId the ID of the User Pool.\nThe keyfunc Go library, has a convenient method keyfunc.Get for retrieving that key using an HTTP request and parsing it, all in a single statement:\nimport ( \u0026#34;fmt\u0026#34; \u0026#34;github.com/MicahParks/keyfunc/v2\u0026#34; ) func GetJWKS(awsRegion string, cognitoUserPoolId string) (*keyfunc.JWKS, error) { jwksURL := fmt.Sprintf(\u0026#34;https://cognito-idp.%s.amazonaws.com/%s/.well-known/jwks.json\u0026#34;, awsRegion, cognitoUserPoolId) jwks, err := keyfunc.Get(jwksURL, keyfunc.Options{}) if err != nil { return nil, err } return jwks, nil } Verifying an incoming JWT in a Gin middleware #1. Defining a Gin authentication middleware #We can define a middleware in a file middlewares/cognito.go, with the following function signature, passing the token use (requiredTokenUse which can be access or id depending on the type of Cognito token we accept), AWS Region (awsDefaultRegion), the Cognito User Pool ID (cognitoUserPoolId), the Cognito App Client ID that we\u0026rsquo;ve created (cognitoAppClientId) and the JWKS we retrieved earlier (jwks).\nfunc CognitoAuthMiddleware(requiredTokenUse string, awsDefaultRegion string, cognitoUserPoolId string, cognitoAppClientId string, jwks *keyfunc.JWKS) gin.HandlerFunc { return func(c *gin.Context) { } } 2. Retrieving the JWT from an incoming request #By convention, an incoming request contains a JWT in its Authorization header using a Bearer token.\nThe Authorization header including the Bearer token has the format:\nAuthorization: Bearer abcdefg Using the Gin context passed to our middleware (variable c), we can retrieve the header, and get the raw value of the JWT by splitting the string containing it.\n// Retrieve JWT from the \u0026#34;Authorization\u0026#34; header authHeader := c.GetHeader(\u0026#34;Authorization\u0026#34;) splitToken := strings.Split(authHeader, \u0026#34;Bearer \u0026#34;) if len(splitToken) != 2 { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } tokenString := splitToken[1] if tokenString == \u0026#34;\u0026#34; { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } 3. Verify the JWT signature, signing algorithm, issuer (iss) and existence of expiry time (exp) #Now, using golang-jwt we can perform some first rudimentary checks against the JWT using some convenient methods offered by the library.\nThe signature of the JWT is verified using the jwt.Parse function by providing the JWKS we retrieved earlier (jwks.Keyfunc), along with some extra checks we define below.\nOne of the most important verifications is defining the specific valid algorithm methods that the parser will use, and confirming that the incoming token is using those exclusively. In the case of Amazon Cognito the asymmetric algorithm RS256 is used to sign the key. This can be enforced using jwt.WithValidMethods. Leaving this option without a value leaves us open to algorithm confusion attacks.\nThe issuer claim (iss) should match our User Pool. For example, a User Pool created in the us-east-1 region will have the following iss value: https://cognito-idp.us-east-1.amazonaws.com/\u0026lt;userpoolID\u0026gt;. We can verify this using the jwt.WithIssuer function.\nWe can also preemptively check that the exp claim defining the expiration time exists in the token (using jwt.WithExpirationRequired). This simply checks that it is present in the token and not its value.\nThe above checks can be succinctly defined in one statement, facilitated by golang-jwt, as followed:\ntoken, err := jwt.Parse(tokenString, jwks.Keyfunc, jwt.WithValidMethods([]string{\u0026#34;RS256\u0026#34;}), jwt.WithExpirationRequired(), jwt.WithIssuer(fmt.Sprintf(\u0026#34;https://cognito-idp.%s.amazonaws.com/%s\u0026#34;, awsDefaultRegion, cognitoUserPoolId))) if err != nil || !token.Valid { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } 4. Parse JWT claims #If the above checks succeeded, we can attempt to parse the token\u0026rsquo;s claims by casting them in a jwt.MapClaims struct.\nclaims, ok := token.Claims.(jwt.MapClaims) if !ok { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } 5. Compare the exp claim to the current time #Now we can start making additional checks against the claims\u0026rsquo; values found in the token, starting with its expiry time (exp claim) and using jwt.GetExpirationTime.\nexpClaim, err := claims.GetExpirationTime() if err != nil { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } if expClaim.Unix() \u0026lt; time.Now().Unix() { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } 6. Check the token_use claim #Cognito can send ID or access tokens, each with a different set of attributes. Depending on the nature of the endpoint we want to protect we can choose to accept specific types.\nID tokens contain claims about the identity of the authenticated user, such as name, email, and phone_number.\nAccess tokens contain claims about the authorization of the authenticated user such as a list of the user\u0026rsquo;s groups, and a list of scopes.\ntokenUseClaim, ok := claims[\u0026#34;token_use\u0026#34;].(string) if !ok { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } if tokenUseClaim != requiredTokenUse { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } 7. Verify the existence of the subject (sub) claim #The subject (sub) claim is mandatory and contains the ID of the Cognito user. We can also set its value in the Gin context, for further processing in the request.\nsubClaim, err := claims.GetSubject() if err != nil { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } c.Set(\u0026#34;username\u0026#34;, subClaim) 8. Verify the audience (aud)/client ID (client_id) claim #Depending on the type of token (access or ID), we can check respectively the aud or the client_id claims and that they should match the Cognito app client ID created in the Cognito User Pool.\nThe aud claim can be retrieved using jwt.GetAudience, the client_id custom claim can be retrieved by manually checking its existence in the JWT claims.\nvar appClientIdClaim string if tokenUseClaim == \u0026#34;id\u0026#34; { audienceClaims, err := claims.GetAudience() if err != nil { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } appClientIdClaim = audienceClaims[0] } else if tokenUseClaim == \u0026#34;access\u0026#34; { clientIdClaim, ok := claims[\u0026#34;client_id\u0026#34;].(string) if !ok { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } appClientIdClaim = clientIdClaim } else { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } if appClientIdClaim != cognitoAppClientId { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } 9. Optionally retrieve any Cognito user groups that the user belongs to #We can optionally parse any Cognito groups that the user belongs to, and set them in the Gin context for further usage within the request. These exist in a cognito:groups claim in the JWT.\nuserGroupsAttribute, ok := claims[\u0026#34;cognito:groups\u0026#34;] userGroupsClaims := make([]string, 0) if ok { switch x := userGroupsAttribute.(type) { case []interface{}: for _, e := range x { userGroupsClaims = append(userGroupsClaims, e.(string)) } default: c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } } c.Set(\u0026#34;groups\u0026#34;, userGroupsClaims) 10. Finally, proceed to the actual request #c.Next() Complete middleware snippet #The complete snippet using all the above statements is as followed:\npackage middlewares import ( \u0026#34;fmt\u0026#34; \u0026#34;net/http\u0026#34; \u0026#34;strings\u0026#34; \u0026#34;time\u0026#34; \u0026#34;github.com/MicahParks/keyfunc/v2\u0026#34; \u0026#34;github.com/gin-gonic/gin\u0026#34; \u0026#34;github.com/golang-jwt/jwt/v5\u0026#34; ) // Gin middleware for verifying an incoming Cognito JWT, embedded in the request headers // https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html func CognitoAuthMiddleware(requiredTokenUse string, awsDefaultRegion string, cognitoUserPoolId string, cognitoAppClientId string, jwks *keyfunc.JWKS) gin.HandlerFunc { return func(c *gin.Context) { // Retrieve JWT from the \u0026#34;Authorization\u0026#34; header authHeader := c.GetHeader(\u0026#34;Authorization\u0026#34;) splitToken := strings.Split(authHeader, \u0026#34;Bearer \u0026#34;) if len(splitToken) != 2 { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } tokenString := splitToken[1] if tokenString == \u0026#34;\u0026#34; { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } // * Verify the signature of the JWT // * Verify that the algorithm used is RS256 // * Verify that the \u0026#39;exp\u0026#39; claim exists in the token // * Verification of audience \u0026#39;aud\u0026#39; is taken care later when we examine if the // token is \u0026#39;id\u0026#39; or \u0026#39;access\u0026#39; // * The issuer (iss) claim should match your user pool. For example, a user // pool created in the us-east-1 region // will have the following iss value: https://cognito-idp.us-east-1.amazonaws.com/\u0026lt;userpoolID\u0026gt;. token, err := jwt.Parse(tokenString, jwks.Keyfunc, jwt.WithValidMethods([]string{\u0026#34;RS256\u0026#34;}), jwt.WithExpirationRequired(), jwt.WithIssuer(fmt.Sprintf(\u0026#34;https://cognito-idp.%s.amazonaws.com/%s\u0026#34;, awsDefaultRegion, cognitoUserPoolId))) if err != nil || !token.Valid { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } // Attempt to parse the JWT claims claims, ok := token.Claims.(jwt.MapClaims) if !ok { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } // Compare the \u0026#34;exp\u0026#34; claim to the current time expClaim, err := claims.GetExpirationTime() if err != nil { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } if expClaim.Unix() \u0026lt; time.Now().Unix() { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } // Check the token_use claim. // If you are only accepting the access token in your web API operations, its value must be access. // If you are only using the ID token, its value must be id. // If you are using both ID and access tokens, the token_use claim must be either id or access. tokenUseClaim, ok := claims[\u0026#34;token_use\u0026#34;].(string) if !ok { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } if tokenUseClaim != requiredTokenUse { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } // \u0026#34;sub\u0026#34; claim exists in both ID and Access tokens subClaim, err := claims.GetSubject() if err != nil { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } c.Set(\u0026#34;username\u0026#34;, subClaim) // The \u0026#34;aud\u0026#34; claim in an ID token and the \u0026#34;client_id\u0026#34; claim in an access token should match the app // client ID that was created in the Amazon Cognito user pool. var appClientIdClaim string if tokenUseClaim == \u0026#34;id\u0026#34; { audienceClaims, err := claims.GetAudience() if err != nil { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } appClientIdClaim = audienceClaims[0] } else if tokenUseClaim == \u0026#34;access\u0026#34; { clientIdClaim, ok := claims[\u0026#34;client_id\u0026#34;].(string) if !ok { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } appClientIdClaim = clientIdClaim } else { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } if appClientIdClaim != cognitoAppClientId { c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } // Retrieve any Cognito user groups that the user belongs to userGroupsAttribute, ok := claims[\u0026#34;cognito:groups\u0026#34;] userGroupsClaims := make([]string, 0) if ok { switch x := userGroupsAttribute.(type) { case []interface{}: for _, e := range x { userGroupsClaims = append(userGroupsClaims, e.(string)) } default: c.JSON(http.StatusUnauthorized, gin.H{\u0026#34;error\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) c.Abort() return } } c.Set(\u0026#34;groups\u0026#34;, userGroupsClaims) c.Next() } } Using the middleware in a Gin endpoint #Putting it all together, we can create Gin endpoints that are protected using the above middleware.\nThe below code shows how to define a Go main function containing two protected Gin endpoints, one requiring a Cognito ID token (/protected-with-id-token) and one requiring a Cognito Access Token (/protected-with-access-token).\npackage main import ( \u0026#34;fmt\u0026#34; \u0026#34;log\u0026#34; \u0026#34;net/http\u0026#34; \u0026#34;os\u0026#34; \u0026#34;github.com/MicahParks/keyfunc/v2\u0026#34; \u0026#34;github.com/angelospanag/go-gin-cognito-jwt-verification/middleware\u0026#34; \u0026#34;github.com/gin-gonic/gin\u0026#34; \u0026#34;github.com/joho/godotenv\u0026#34; ) func GetJWKS(awsRegion string, cognitoUserPoolId string) (*keyfunc.JWKS, error) { jwksURL := fmt.Sprintf(\u0026#34;https://cognito-idp.%s.amazonaws.com/%s/.well-known/jwks.json\u0026#34;, awsRegion, cognitoUserPoolId) jwks, err := keyfunc.Get(jwksURL, keyfunc.Options{}) if err != nil { return nil, err } return jwks, nil } func main() { err := godotenv.Load() if err != nil { log.Fatal(\u0026#34;Error loading .env file\u0026#34;) } awsDefaultRegion := os.Getenv(\u0026#34;AWS_DEFAULT_REGION\u0026#34;) cognitoUserPoolId := os.Getenv(\u0026#34;COGNITO_USER_POOL_ID\u0026#34;) cognitoAppClientId := os.Getenv(\u0026#34;COGNITO_APP_CLIENT_ID\u0026#34;) jwks, err := GetJWKS(awsDefaultRegion, cognitoUserPoolId) if err != nil { log.Fatalf(\u0026#34;Failed to retrieve Cognito JWKS\\nError: %s\u0026#34;, err) } router := gin.Default() router.GET(\u0026#34;/healthcheck\u0026#34;, func(c *gin.Context) { c.String(http.StatusOK, \u0026#34;ok\u0026#34;) }) router.GET(\u0026#34;/protected-with-id-token\u0026#34;, middlewares.CognitoAuthMiddleware( \u0026#34;id\u0026#34;, awsDefaultRegion, cognitoUserPoolId, cognitoAppClientId, jwks), func(c *gin.Context) { username, _ := c.Get(\u0026#34;username\u0026#34;) c.JSON(http.StatusOK, gin.H{\u0026#34;username\u0026#34;: username}) }) router.GET(\u0026#34;/protected-with-access-token\u0026#34;, middlewares.CognitoAuthMiddleware( \u0026#34;access\u0026#34;, awsDefaultRegion, cognitoUserPoolId, cognitoAppClientId, jwks), func(c *gin.Context) { username, _ := c.Get(\u0026#34;username\u0026#34;) c.JSON(http.StatusOK, gin.H{\u0026#34;username\u0026#34;: username}) }) router.Run() } Full implementation #You can find the fully working implementation in a GitHub repository.\n","date":"3 December 2023","permalink":"https://www.angelospanag.me/posts/verifying-a-json-web-token-from-cognito-in-go-and-gin/","section":"Posts","summary":"How to secure a Go backend using Amazon Cognito","title":"Verifying a JSON Web Token from Amazon Cognito in Go and Gin"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/bun/","section":"Tags","summary":"","title":"Bun"},{"content":" Intro Installing Bun and scaffolding a new Next.js project Containerising our new Next.js project Full Dockerfile Building the Docker image and running it as a container Intro #Bun is a new performant JavaScript runtime and toolkit containing its own bundler, test runner and package manager, while aiming to be a drop-in replacement for existing projects using Node.js. At the time of writing it has reached version 1.0.2, and I had the chance to test it with some personal and work projects, including using it in combination with Next.js in Docker containers.\nSo far I have been really impressed with its performance and its compatibility with existing projects. Below I am showing you how to containerise a Next.js project using Bun, with minimal changes from the official documentation.\nInstalling Bun and scaffolding a new Next.js project #The official documentation offers many alternatives on how to install Bun, the quickest is:\ncurl -fsSL https://bun.sh/install | bash After that, creating a new Next.js project is as easy as:\nbunx create-next-app Containerising our new Next.js project #Vercel offers guidelines in their official documentation on how to create a Docker image along with a sample repository.\nFirst we need to modify the next.config.js file so that the build output can be standalone, as shown in the official documentation.\n// next.config.js module.exports = { // ... rest of the configuration. output: \u0026#39;standalone\u0026#39;, } After that, we can reuse the existing example by making the following modifications in the provided Dockerfile:\nThe official base image is defined as FROM oven/bun AS base and is released by Oven, \u0026ldquo;the company behind Bun\u0026rdquo; in DockerHub.\nBun is using by default a binary lockfile bun.lockb for performance. In order to install a project\u0026rsquo;s dependencies we need to copy first the package.json and the generated bun.lockb lockfile.\nCOPY package.json bun.lockb ./ Installing using reproducible dependencies can be done with: bun install --frozen-lockfile\nRUN bun install --frozen-lockfile The build step can be run using: bun run build\nRUN bun run build The base image is creating a unix group called bun. We don\u0026rsquo;t need to create a new separate group nodejs as it was done in the previous version of the Dockerfile. We only need to change the ownership of the generated files from the build step to nextjs:bun.\nRUN chown nextjs:bun .next COPY --from=builder --chown=nextjs:bun /app/.next/standalone ./ COPY --from=builder --chown=nextjs:bun /app/.next/static ./.next/static Finally, the server can be run with: bun server.js\nCMD [\u0026#34;bun\u0026#34;, \u0026#34;server.js\u0026#34;] Optionally, my personal preference is to disable the collection of anonymous telemetry during the build step and for the final produced Docker image.\nENV NEXT_TELEMETRY_DISABLED 1 Full Dockerfile #The full Dockerfile with the above changes can now be added to the root of the Next.js project and is as followed:\nFROM oven/bun AS base # Install dependencies only when needed FROM base AS deps WORKDIR /app # Install dependencies COPY package.json bun.lockb ./ RUN bun install --frozen-lockfile # Rebuild the source code only when needed FROM base AS builder WORKDIR /app COPY --from=deps /app/node_modules ./node_modules COPY . . # Next.js collects completely anonymous telemetry data about general usage. # Learn more here: https://nextjs.org/telemetry # Disable telemetry during the build ENV NEXT_TELEMETRY_DISABLED 1 RUN bun run build # Production image, copy all the files and run next FROM base AS runner WORKDIR /app ENV NODE_ENV production # Disable telemetry ENV NEXT_TELEMETRY_DISABLED 1 RUN adduser --system --uid 1001 nextjs COPY --from=builder /app/public ./public # Set the correct permission for prerender cache RUN mkdir .next RUN chown nextjs:bun .next # Automatically leverage output traces to reduce image size # https://nextjs.org/docs/advanced-features/output-file-tracing COPY --from=builder --chown=nextjs:bun /app/.next/standalone ./ COPY --from=builder --chown=nextjs:bun /app/.next/static ./.next/static USER nextjs EXPOSE 3000 ENV PORT 3000 # Set hostname to localhost ENV HOSTNAME \u0026#34;0.0.0.0\u0026#34; CMD [\u0026#34;bun\u0026#34;, \u0026#34;server.js\u0026#34;] Building the Docker image and running it as a container #As before, the image can be built and tagged using:\ndocker build -t my-app . Finally, running a container of the image:\ndocker run -p 3000:3000 my-app ","date":"17 September 2023","permalink":"https://www.angelospanag.me/posts/containerising-nextjs-using-docker-and-bun/","section":"Posts","summary":"Using the newest JavaScript runtime with Next.js and Docker","title":"Containerising Next.js using Docker and Bun"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/docker/","section":"Tags","summary":"","title":"Docker"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/javascript/","section":"Tags","summary":"","title":"Javascript"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/nextjs/","section":"Tags","summary":"","title":"Nextjs"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/typescript/","section":"Tags","summary":"","title":"Typescript"},{"content":" Intro Setting up an example FastAPI app that serves templates containing HTMX Why this setup could go wrong How can this be avoided Conclusion Intro #HTMX has become one of my favourite minimalistic frontend libraries to use in combination with a FastAPI/Python project. The JavaScript/React fatigue is real.\nThis post will not be introducing HTMX from scratch, but rather mention an elegant way to avoid a common unwanted behaviour that you might face in your application, the accidental loading of partial pages which are normally intended as responses to an HTMX request.\nSetting up an example FastAPI app that serves templates containing HTMX #Supposedly you have a FastAPI backend that serves you a basic HTML page with some minimal CSS, a header \u0026ldquo;Sci-Fi Movies\u0026rdquo; and a button \u0026ldquo;Get movies table\u0026rdquo;. A pretty basic page.\nMovies initial page The HTML in templates/index.html looks as followed:\n\u0026lt;!DOCTYPE html\u0026gt; \u0026lt;html lang=\u0026#34;en\u0026#34;\u0026gt; \u0026lt;head\u0026gt; \u0026lt;meta charset=\u0026#34;UTF-8\u0026#34; /\u0026gt; \u0026lt;script src=\u0026#34;https://unpkg.com/htmx.org@1.8.6\u0026#34;\u0026gt;\u0026lt;/script\u0026gt; \u0026lt;style\u0026gt; body { background-color: #222; color: #fff; font-family: Arial, sans-serif; } table { border-collapse: collapse; margin: 20px auto; width: 80%; } th, td { padding: 10px; text-align: left; border-bottom: 1px solid #ddd; } th { background-color: #333; } tr:nth-child(even) { background-color: #444; } \u0026lt;/style\u0026gt; \u0026lt;/head\u0026gt; \u0026lt;body\u0026gt; \u0026lt;h1\u0026gt;Sci-Fi Movies\u0026lt;/h1\u0026gt; \u0026lt;button hx-get=\u0026#34;/movies\u0026#34; hx-trigger=\u0026#34;click\u0026#34; hx-target=\u0026#34;#movies\u0026#34; hx-swap=\u0026#34;outerHTML\u0026#34;\u0026gt; Get movies table \u0026lt;/button\u0026gt; \u0026lt;div id=\u0026#34;movies\u0026#34;\u0026gt;\u0026lt;/div\u0026gt; \u0026lt;/body\u0026gt; \u0026lt;/html\u0026gt; The actual movies table can reside in an HTML file templates/movies.html as followed:\n\u0026lt;table\u0026gt; \u0026lt;thead\u0026gt; \u0026lt;tr\u0026gt; \u0026lt;th\u0026gt;Title\u0026lt;/th\u0026gt; \u0026lt;th\u0026gt;Director\u0026lt;/th\u0026gt; \u0026lt;th\u0026gt;Release Year\u0026lt;/th\u0026gt; \u0026lt;th\u0026gt;IMDb Rating\u0026lt;/th\u0026gt; \u0026lt;/tr\u0026gt; \u0026lt;/thead\u0026gt; \u0026lt;tbody\u0026gt; \u0026lt;tr\u0026gt; \u0026lt;td\u0026gt;Blade Runner\u0026lt;/td\u0026gt; \u0026lt;td\u0026gt;Ridley Scott\u0026lt;/td\u0026gt; \u0026lt;td\u0026gt;1982\u0026lt;/td\u0026gt; \u0026lt;td\u0026gt;8.1\u0026lt;/td\u0026gt; \u0026lt;/tr\u0026gt; \u0026lt;tr\u0026gt; \u0026lt;td\u0026gt;The Matrix\u0026lt;/td\u0026gt; \u0026lt;td\u0026gt;The Wachowskis\u0026lt;/td\u0026gt; \u0026lt;td\u0026gt;1999\u0026lt;/td\u0026gt; \u0026lt;td\u0026gt;8.7\u0026lt;/td\u0026gt; \u0026lt;/tr\u0026gt; \u0026lt;tr\u0026gt; \u0026lt;td\u0026gt;Inception\u0026lt;/td\u0026gt; \u0026lt;td\u0026gt;Christopher Nolan\u0026lt;/td\u0026gt; \u0026lt;td\u0026gt;2010\u0026lt;/td\u0026gt; \u0026lt;td\u0026gt;8.8\u0026lt;/td\u0026gt; \u0026lt;/tr\u0026gt; \u0026lt;tr\u0026gt; \u0026lt;td\u0026gt;Interstellar\u0026lt;/td\u0026gt; \u0026lt;td\u0026gt;Christopher Nolan\u0026lt;/td\u0026gt; \u0026lt;td\u0026gt;2014\u0026lt;/td\u0026gt; \u0026lt;td\u0026gt;8.6\u0026lt;/td\u0026gt; \u0026lt;/tr\u0026gt; \u0026lt;tr\u0026gt; \u0026lt;td\u0026gt;Avatar\u0026lt;/td\u0026gt; \u0026lt;td\u0026gt;James Cameron\u0026lt;/td\u0026gt; \u0026lt;td\u0026gt;2009\u0026lt;/td\u0026gt; \u0026lt;td\u0026gt;7.8\u0026lt;/td\u0026gt; \u0026lt;/tr\u0026gt; \u0026lt;/tbody\u0026gt; \u0026lt;/table\u0026gt; When the button is clicked, an HTMX GET call is performed against /movies which will fetch an HTML response containing movies rendered in a table, and replace the empty div with id movies. A very simple example that showcases the flexibility and terse syntax of HTMX.\n\u0026lt;button hx-get=\u0026#34;/movies\u0026#34; hx-trigger=\u0026#34;click\u0026#34; hx-target=\u0026#34;#movies\u0026#34; hx-swap=\u0026#34;outerHTML\u0026#34;\u0026gt; Get movies table \u0026lt;/button\u0026gt; \u0026lt;div id=\u0026#34;movies\u0026#34;\u0026gt;\u0026lt;/div\u0026gt; The backend side of what we have done so far can be in a main.py file that looks like this:\nfrom fastapi import Depends, FastAPI, HTTPException, Query, Request, status from fastapi.responses import HTMLResponse from fastapi.staticfiles import StaticFiles from fastapi.templating import Jinja2Templates app = FastAPI() templates = Jinja2Templates(directory=\u0026#34;templates\u0026#34;) @app.get(\u0026#34;/movies\u0026#34;, response_class=HTMLResponse) async def get_movies(request: Request): return templates.TemplateResponse(\u0026#34;movies.html\u0026#34;, {\u0026#34;request\u0026#34;: request}) @app.get(\u0026#34;/\u0026#34;, response_class=HTMLResponse) async def index(request: Request): return templates.TemplateResponse(\u0026#34;index.html\u0026#34;, {\u0026#34;request\u0026#34;: request}) The app works as intended. If we visit http://localhost:8000/ and click the Get movies table button, we see the table loaded on the screen while the CSS styling is respected according to the rules defined in index.html.\nMovies table HTML partial loaded correctly with HTMX Why this setup could go wrong #If a user accidentally visits http://localhost:8000/movies then the template movies.html will load, which simply contains our raw HTML table with no styles defined. This page is effectively a partial that should only be loaded as a result of an HTMX call, not on its own.\nMovies table HTML partial loaded with no styling How can this be avoided #Regardless of your backend solution, you can check that a request is coming from HTMX by checking for the existence of the hx-request header: {'hx-request': 'true'}. FastAPI offers a very elegant dependency injection system which we can leverage for this exact purpose.\nWe can define a function is_partial_rendering which accepts the Request object of an HTTP call, accesses its headers, checks that a header {'hx-request': 'true'} exists, and continues with serving the request normally, otherwise it redirects the user with an HTTP status 307 to the root / endpoint of the FastAPI app.\nThat dependency in turn can be injected to our /movies endpoint as an argument with _=Depends(is_partial_rendering).\nasync def is_partial_rendering(request: Request): if \u0026#34;hx-request\u0026#34; in request.headers: if not request.headers[\u0026#34;hx-request\u0026#34;]: raise HTTPException( status_code=status.HTTP_307_TEMPORARY_REDIRECT, headers={\u0026#34;Location\u0026#34;: \u0026#34;/\u0026#34;}, ) else: raise HTTPException( status_code=status.HTTP_307_TEMPORARY_REDIRECT, headers={\u0026#34;Location\u0026#34;: \u0026#34;/\u0026#34;}, ) @app.get(\u0026#34;/movies\u0026#34;, response_class=HTMLResponse) async def get_movies( request: Request, _=Depends(is_partial_rendering), ): return templates.TemplateResponse(\u0026#34;movies.html\u0026#34;, {\u0026#34;request\u0026#34;: request}) Conclusion #A very simple but crucial trick that leverages Dependencies, one of the most useful features of FastAPI, and can be used to make a frontend codebase using HTMX more robust.\nYou can view the final result of this post as a gist here.\n","date":"3 April 2023","permalink":"https://www.angelospanag.me/posts/fastapi-with-htmx-partials/","section":"Posts","summary":"Avoid accidental page loads of HTMX partials in your FastAPI app","title":"FastAPI with HTMX partials"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/htmx/","section":"Tags","summary":"","title":"Htmx"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/logging/","section":"Tags","summary":"","title":"Logging"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/structlog/","section":"Tags","summary":"","title":"Structlog"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/structured-logging/","section":"Tags","summary":"","title":"Structured Logging"},{"content":" Intro Integrating structlog with FastAPI Configuring structlog in a FastAPI app Adding logging variables to every request Logging context variables across different layers of the app Testing with an ID that exists Testing with an ID that does not exist Load balancing and health checks Conclusion Intro #I have gradually shifted to using and promoting the usage of structured logging to all the projects I am being involved with. Essentially, instead of using raw logs, you opt for logging events associated with key-pair values, while at the same time you can bind context variables that are present to every subsequent logging call.\nBoth features are extremely useful in the context of a backend web app, where each request might span multiple layers of functionality, from a view all the way to a database layer. Even more so, if you would want useful metrics from your logs about for how your app behaves and is used from your users. Structured logging greatly facilitates the parsing and aggregation of those logs.\nFor that purpose I have been using structlog, one of the most flexible and powerful Python packages that performs structured logging, in combination with my favourite web framework FastAPI.\nBorrowing the simplest official examples from structlog, each log entry now is an event with key-pair values that contain easily parsable information:\nfrom structlog import get_logger log = get_logger() log.info(\u0026#34;key_value_logging\u0026#34;, out_of_the_box=True, effort=0) # Output # 2020-11-18 09:17:09 [info] key_value_logging effort=0 out_of_the_box=True Combined with the binding of context variables to threads, this functionality is invaluable for doing proper logging across multiple layers in your app:\nlog = log.bind(user=\u0026#34;anonymous\u0026#34;, some_key=23) log = log.bind(user=\u0026#34;hynek\u0026#34;, another_key=42) log.info(\u0026#34;user.logged_in\u0026#34;, happy=True) # Output # 2020-11-18 09:18:28 [info] user.logged_in another_key=42 happy=True some_key=23 user=hynek Integrating structlog with FastAPI #For the purposes of this guide I am keeping all my code in an app folder at the root of the project and using Poetry to install the necessary dependencies:\nmkdir fastapi-structlog cd fastapi-structlog poetry init poetry add fastapi uvicorn[standard] poetry add ruff black --group dev # Optional linting and formatting libraries mkdir app Configuring structlog in a FastAPI app #structlog configuration should be initialised as early as possible in the lifetime of an app. A common pattern I have used is to include it in an __init__.py file located in the same module where the FastAPI initialiser is located. In that way, despite the fact that we will be starting our server by targeting the FastAPI initialiser in a different file in the same module, the below statements that configure structlog will always be executed first.\nA minimal src/__init__.py I use, looks like the following:\nimport logging import sys import structlog log = structlog.get_logger() # Disable uvicorn logging logging.getLogger(\u0026#34;uvicorn.error\u0026#34;).disabled = True logging.getLogger(\u0026#34;uvicorn.access\u0026#34;).disabled = True # Structlog configuration logging.basicConfig(format=\u0026#34;%(message)s\u0026#34;, stream=sys.stdout, level=logging.INFO) structlog.configure( processors=[ structlog.contextvars.merge_contextvars, structlog.processors.add_log_level, structlog.processors.StackInfoRenderer(), structlog.dev.set_exc_info, structlog.processors.TimeStamper(fmt=\u0026#34;iso\u0026#34;, utc=True), structlog.dev.ConsoleRenderer(), ], logger_factory=structlog.PrintLoggerFactory(), ) In order to avoid duplicate logs produced by the uvicorn server, I usually opt to disable its dedicated loggers uvicorn.error and uvicorn.access.\nThe rest of the configuration are common processors used in structlog, among the many found in its API:\nstructlog.contextvars.merge_contextvars - merges any key-pairs found in a global context structlog.processors.add_log_level - Adds the log level to the event dict under the level key. structlog.processors.StackInfoRenderer() - Allows you to add a stack dump to log entries structlog.processors.TimeStamper(fmt=\u0026quot;iso\u0026quot;, utc=True) - Adds timestamps using the ISO 8601 international standard structlog.dev.ConsoleRenderer() - Uses a console renderer I often switch the ConsoleRender with a JSONRenderer so all logs are produced in JSON format for easier parsing.\nWe can now create a src/main.py containing the FastAPI initialiser:\nimport structlog from fastapi import FastAPI app = FastAPI() logger = structlog.get_logger() Finally, we can run the uvicorn server from the root of the project, specifying the FastAPI initialiser in main.py:\nuvicorn src.main:app --reload Adding logging variables to every request #Our server runs but it doesn\u0026rsquo;t do much. For now, we haven\u0026rsquo;t specified any endpoints or a way for structlog to automatically create logs with each endpoint call.\nIn order to do that we can take advantage of a FastAPI middleware, essentially functionality that will always run before a request is processed and before returning a response.\nsrc/main.py\nimport structlog from fastapi import FastAPI, HTTPException, Request, Response @app.middleware(\u0026#34;http\u0026#34;) async def logger_middleware(request: Request, call_next): # Clear previous context variables structlog.contextvars.clear_contextvars() # Bind new variables identifying the request and a generated UUID structlog.contextvars.bind_contextvars( path=request.url.path, method=request.method, client_host=request.client.host, request_id=str(uuid.uuid4()), ) # Make the request and receive a response response = await call_next(request) # Bind the status code of the response structlog.contextvars.bind_contextvars( status_code=response.status_code, ) if 400 \u0026lt;= response.status_code \u0026lt; 500: logger.warn(\u0026#34;Client error\u0026#34;) elif response.status_code \u0026gt;= 500: logger.error(\u0026#34;Server error\u0026#34;) else: logger.info(\u0026#34;OK\u0026#34;) return response Here we see the first usage the aforementioned context variables. This middleware is the point where we will be binding variables that will be included in every subsequent logging call of our web request.\nInitially we clear any context variables previously found the same thread. Then, in our example we use the following values taken from the FastAPI Request object:\nthe URL path (request.url.path) of the request the request method (request.method) the IP of the client that initiated the request (request.client.host) We can also add a unique identifier to our web request (uuid.uuid4()). The usage of a UUID is ideal for this. By generating a unique identifier attached to every log produced from the same web request, we can easily track all actions taken within its lifetime by filtering our logs for that identifier, extremely useful in combination with logging aggregators like Cloudwatch or Elasticsearch/Logstash.\nAfter our request is processed, we can add the status code of the response take from the FastAPI response object (response.status_code). That\u0026rsquo;s why I usually add some checks at the end of the middleware to output warning level logs for any 4** status codes, error level logs for any 5** status codes and info level logs for the rest.\nLogging context variables across different layers of the app #We can see this in action by creating a \u0026ldquo;database\u0026rdquo; layer in our app existing in a different file that gets called by the view layer of our app. All logs across those layers will hold the context logging variables (path, method, client_host) that we have specified in the above FastAPI middleware, along with a UUID (request_id).\nFor simplicity, we can emulate a database as a Python list that contains dictionaries of items. We can then define get_item function that will return an item\u0026rsquo;s details given its ID, otherwise it returns None.\nimport structlog log = structlog.get_logger() items: list[dict] = [{\u0026#34;id\u0026#34;: 1, \u0026#34;name\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;availability\u0026#34;: 2}] def get_item(item_id: int) -\u0026gt; dict | None: for item in items: if item[\u0026#34;id\u0026#34;] == item_id: log.info(\u0026#34;Item found\u0026#34;, item_id=item_id) return item log.warn(\u0026#34;Item not found\u0026#34;, item_id=item_id) return None We can now make use of the get_item function defined in the \u0026ldquo;database layer\u0026rdquo; and create an endpoint that accepts an ID as a path parameter. If the item exists in our database, we return it as a JSON with a status code 200, otherwise we return a message \u0026ldquo;Item not found\u0026rdquo; with a status code 404.\nAlso, by taking advantage of the integration between Pydantic and FastAPI, we can create a schema of an item as a Pydantic model and set this as the response model in our new endpoint:\nsrc/schemas.py\nfrom pydantic import BaseModel class Item(BaseModel): id: int name: str availability: int src/main.py\nfrom src.db import get_item from src.schemas import Item @app.get(\u0026#34;/items/{item_id}\u0026#34;, response_model=Item) async def root(item_id: int): logger.info(\u0026#34;Retrieving item\u0026#34;) item: dict | None = get_item(item_id) if not item: raise HTTPException(status_code=404, detail=\u0026#34;Item not found\u0026#34;) return item Testing with an ID that exists #curl http://localhost:8000/items/1 { \u0026#34;id\u0026#34;: 1, \u0026#34;name\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;availability\u0026#34;: 2 } 2023-01-07T12:04:47.650855Z [info] Retrieving item client_host=127.0.0.1 method=GET path=/items/1 request_id=39028224-5807-4f3f-86e1-f4b55786510d 2023-01-07T12:04:47.651002Z [info] Item found client_host=127.0.0.1 item_id=1 method=GET path=/items/1 request_id=39028224-5807-4f3f-86e1-f4b55786510d 2023-01-07T12:04:47.651489Z [info] OK client_host=127.0.0.1 method=GET path=/items/1 request_id=39028224-5807-4f3f-86e1-f4b55786510d status_code=200 Notice how our logs associated with the request, contain the same request_id (UUID), path, method and client_host generated from the FastAPI middleware we defined above, even if they are generated from different files within the app. The last log contains the status_code added after our request is processed, and we know its value from the response object.\nTesting with an ID that does not exist #We can see similar results by testing with an ID is not found in our database:\ncurl http://localhost:8000/items/2 { \u0026#34;detail\u0026#34;: \u0026#34;Item not found\u0026#34; } 2023-01-07T12:05:55.532626Z [info] Retrieving item client_host=127.0.0.1 method=GET path=/items/2 request_id=fa15a357-2a93-459f-b87d-0bff2b0cc7b1 2023-01-07T12:05:55.532750Z [warning] Item not found client_host=127.0.0.1 item_id=2 method=GET path=/items/2 request_id=fa15a357-2a93-459f-b87d-0bff2b0cc7b1 2023-01-07T12:05:55.533241Z [warning] Client error client_host=127.0.0.1 method=GET path=/items/2 request_id=fa15a357-2a93-459f-b87d-0bff2b0cc7b1 status_code=404 Load balancing and health checks #If you are using any form of load balancing that relies on an HTTP healthcheck endpoint to check the health of a distributed service by querying it periodically (for example AWS Application Load Balancer or Kubernetes load balancing), your logs will be spammed with the same entry that shows that endpoint being called every few seconds.\nGiven a simple healthcheck FastAPI endpoint that returns a 200:\n@app.get(\u0026#34;/healthcheck\u0026#34;) async def healthcheck(): return Response() Then the logs would probably look as followed:\n2000-01-01T00:00:00.000000Z [info] OK client_host=127.0.0.1 method=GET request_id=d5bdb5e7-e2eb-4453-9f31-09332d851998 status_code=200 view=/healthcheck 2000-01-01T00:00:10.000000Z [info] OK client_host=127.0.0.1 method=GET request_id=5247b752-37ca-4a65-ac93-34123a86d9fd status_code=200 view=/healthcheck 2000-01-01T00:00:20.000000Z [info] OK client_host=127.0.0.1 method=GET request_id=f68233a3-1c1a-4511-9274-ddacadc992a3 status_code=200 view=/healthcheck 2000-01-01T00:00:30.000000Z [info] OK client_host=127.0.0.1 method=GET request_id=f2b51d29-f16d-41ca-b59b-2ac97fcaad13 status_code=200 view=/healthcheck In order to avoid that, I always add a check in the aforementioned FastAPI middleware that tests if the requested path is /healtcheck and excludes it from producing logs. The rest of the middleware functionality remains as is:\nsrc/main.py\nimport structlog from fastapi import FastAPI, HTTPException, Request, Response @app.middleware(\u0026#34;http\u0026#34;) async def logger_middleware(request: Request, call_next): structlog.contextvars.clear_contextvars() structlog.contextvars.bind_contextvars( path=request.url.path, method=request.method, client_host=request.client.host, request_id=str(uuid.uuid4()), ) response = await call_next(request) structlog.contextvars.bind_contextvars( status_code=response.status_code, ) # Exclude /healthcheck endpoint from producing logs if request.url.path != \u0026#34;/healthcheck\u0026#34;: if 400 \u0026lt;= response.status_code \u0026lt; 500: logger.warn(\u0026#34;Client error\u0026#34;) elif response.status_code \u0026gt;= 500: logger.error(\u0026#34;Server error\u0026#34;) else: logger.info(\u0026#34;OK\u0026#34;) return response Conclusion #Investing on using structured logging and abundant logging statements, even from early during development, is one of the most useful practices that can provide clarity on how your application works and produce easily parsable metrics with minimal effort.\nFor that reason, I have included all the above-mentioned code in a starter project I maintain on my GitHub that I use for my personal projects, or as a point of reference to showcase to my colleagues. If you found this useful, feel free to use it for your own projects.\n","date":"5 January 2023","permalink":"https://www.angelospanag.me/posts/structured-logging-using-structlog-and-fastapi/","section":"Posts","summary":"Useful tips for adding structured logging to a FastAPI backend","title":"Structured logging using structlog and FastAPI"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/containerisation/","section":"Tags","summary":"","title":"Containerisation"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/podman/","section":"Tags","summary":"","title":"Podman"},{"content":" Intro Creating a FastAPI app and containerising it with Podman Creating a basic FastAPI skeleton app and sample endpoint Creating a Containerfile/Dockerfile for FastAPI Using podman commands to build and run our app in a container Build a Podman image Running a container Testing and logging Summary Intro #Podman has recently released a desktop client for managing containers. For anyone not familiar with it, Podman is a free and opensource engine for managing and running OCI containers, based on open standards, while aiming to keep compatibility with Docker commands.\nAs their official documentation mentions:\nSimply put: alias docker=podman\n\u0026hellip;and your Docker container workflow should work with no changes.\nCreating a FastAPI app and containerising it with Podman #I wanted to put that to the test using my favourite web framework during the last years, written in Python: FastAPI.\nCreating a basic FastAPI skeleton app and sample endpoint #Starting simple and creating a basic skeleton for our app, while initialising dependency management with another one of my favourite tools, Poetry:\nmkdir fastapi-podman cd fastapi-podman poetry init We add the necessary runtime dependencies, FastAPI for the web framework and uvicorn for the ASGI server:\npoetry add fastapi uvicorn[standard] Last but not least, adding my minimum go-to linting and formatting libraries as development dependencies:\npoetry add flake8 pep8-naming black --group dev Next we create a very simple project skeleton, an app folder that for simplicity will contain all our Python code as a module:\nmkdir app touch app/__init__.py touch app/main.py In our main.py we can now create a simple FastAPI GET endpoint in root that will return a \u0026ldquo;Hello World\u0026rdquo; message in JSON format.\nfrom fastapi import FastAPI app = FastAPI() @app.get(\u0026#34;/\u0026#34;) def read_root(): return {\u0026#34;Hello\u0026#34;: \u0026#34;World\u0026#34;} We can now run the app using uvicorn by pointing to the FastAPI entrypoint:\nuvicorn app.main:app --reload Finally, we can test if our defined endpoint works correctly:\ncurl -i localhost:8000 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 17 100 17 0 0 2990 0 --:--:-- --:--:-- --:--:-- 4250HTTP/1.1 200 OK date: Thu, 10 Nov 2022 14:34:22 GMT server: uvicorn content-length: 17 content-type: application/json {\u0026#34;Hello\u0026#34;:\u0026#34;World\u0026#34;} Creating a Containerfile/Dockerfile for FastAPI #Inspired by the official example from the FastAPI documentation, we can use Docker multi-stage building for creating our final image. The first stage aliased as requirements-stage will install Poetry, copy the Poetry files that track the project\u0026rsquo;s dependencies defined in pyproject.toml and locked in poetry.lock, and finally generate a requirements.txt.\nThe second stage, which will be the one that our container will use to run, will use this generated requirements.txt to install the Python dependencies, copy the required Python project files, and finally run our server with uvicorn.\nNotice how the Dockerfile syntax remains the same (WORKDIR, COPY, RUN etc.) and we can even use the same tags (FROM python:3.11) for referencing the official Python 3.11 image currently available in DockerHub. Podman will fetch it and build it locally with no problems.\nFROM python:3.11 as requirements-stage WORKDIR /tmp RUN pip install poetry COPY ./pyproject.toml ./poetry.lock* /tmp/ RUN poetry export -f requirements.txt --output requirements.txt --without-hashes FROM python:3.11 WORKDIR /code COPY --from=requirements-stage /tmp/requirements.txt /code/requirements.txt RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt COPY ./app /code/app CMD [\u0026#34;uvicorn\u0026#34;, \u0026#34;app.main:app\u0026#34;, \u0026#34;--host\u0026#34;, \u0026#34;0.0.0.0\u0026#34;, \u0026#34;--port\u0026#34;, \u0026#34;80\u0026#34;] Using podman commands to build and run our app in a container #Build a Podman image #The following command will build an image from a Dockerfile in the same directory that it runs from (. character) and tag it as fastapi-podman. Notice how it is exactly the same as the Docker command we would use, the only difference is that we replace docker with podman.\npodman image build -t fastapi-podman . Alternatively, if you prefer to do the same using Podman Desktop and point to the Dockerfile (or at this point, more preferably Containerfile?):\n[Podman Desktop dialog \u0026ldquo;Build image from Containerfile\u0026rdquo; Our images now show up in the Images menu. For example, we see docker.io/library/python which corresponds to the official Python 3.11 image we used as a base for our Dockerfile, \u0026lt;none\u0026gt; which is the intermediate requirements-stage image we built above and finally localhost/fastapi-podman which corresponds to the final image that we will run as a container.\nPodman Desktop showing downloaded and built images: Python 3.11, intermediate requirements-stage and final fastapi-podman Running a container #podman container run -d --name fastapi-podman -p 8000:80 --network bridge fastapi-podman Again if you are familiar with Docker commands, there is complete parity with its Podman equivalent, the flags remain the same:\n-d - run the container as a daemon --name fastapi-podman - assign a name to the container -p 8000:80- map the local port 8000 to the container port 80 --network bridge - run the container in a bridge network fastapi-podman - point to the previously tagged image Alternatively, you can create a container from a Dockerfile/Containerfile or from an existing image in the local registry:\nPodman Desktop dialog \u0026ldquo;Create container from Containerfile\u0026rdquo; We can always check our running containers from the command line:\npodman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a2ff966fc6fd localhost/fastapi-podman:latest uvicorn app.main:... 10 hours ago Up 18 minutes ago 0.0.0.0:8000-\u0026gt;80/tcp fastapi-podman Or using the desktop client in the Containers dialog:\nPodman Desktop showing a running FastAPI container Testing and logging #Performing the same request to the same endpoint as before (curl localhost:8000) gives us back the same JSON response. We can also follow the output of the standard logging of the running app that the container produces, using similar commands to Docker:\npodman logs -f fastapi-podman INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:80 (Press CTRL+C to quit) INFO: Uvicorn running on http://0.0.0.0:80 (Press CTRL+C to quit) INFO: 10.88.0.2:57182 - \u0026#34;GET / HTTP/1.1\u0026#34; 200 OK INFO: 10.88.0.2:57182 - \u0026#34;GET /favicon.ico HTTP/1.1\u0026#34; 404 Not Found \u0026hellip;or again through the desktop client itself:\nPodman Desktop showing FastAPI container logs Summary #Impressive work by the Podman team, so far I had a very positive experience using their CLI and desktop client. I am definitely looking forward to use Podman in more projects.\nYou can find the full example containing the code used in this post, along with instructions on how to install and run Podman with a comprehensive README in a repository on my personal GitHub.\n","date":"10 November 2022","permalink":"https://www.angelospanag.me/posts/podman-an-excellent-alternative-to-docker/","section":"Posts","summary":"Using Podman and Podman Desktop to containerise and run a FastAPI/Python web app","title":"Podman - An excellent alternative to Docker"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/poetry/","section":"Tags","summary":"","title":"Poetry"},{"content":" Intro Advantages of Poetry Creation and management of virtual environments Separation between runtime and development dependencies Version constraints Sub dependencies and conflict resolution Lockfiles Use Poetry with different Python versions Export dependencies to requirements.txt files Bonus items Final thoughts Intro #Over the latest years there have been many dependency management solutions for developing a Python application, starting with the tooling provided from the standard library (pip, easyinstall, venv module) and moving to convenient wrappers like virtualenv, virtualenvwrapper,pip-tools and more recently pipenv and Poetry. Additionally, the Python SoftwareFoundation has made some efforts to introduce a new dependency specification (the latest proposal at this time of writing seems to be PEP 631).\nIt sounds unmanageable to keep track of all this tooling and I apologise if I missed any. In fact, you will find many blogs that touch on the same subject and are referencing XKCD, one of the most popular webcomics which has made fun of this very situation \u0026hellip;because of course \u0026rsquo;there is always a relevant XKCD'.\nMany points can be made about each one and its approach but one of the most popular (and my personal favourite which I have used successfully during the latest years) has been Poetry.\nAdvantages of Poetry #Below are some features of Poetry that have helped me in my everyday work with Python. This is not an exhaustive list by far but its documentation is excellent for anyone who wants to go into more detail.\nAnyone with a JavaScript/npm or PHP/composer background will notice many similarities that have been already existing for their language environment and that have been sorely missed from Python and pip for a long time.\nAssuming you have a working Python on your machine, the most straightforward way you can install it is:\ncurl -sSL https://install.python-poetry.org | python3 - or if you are using brew on macOS:\nbrew install poetry Creation and management of virtual environments #As mentioned above, you can always initialise and manage virtual environments with the Python standard library or with any abstractions on top of it (like virtualenv, virtualenvwrapper). Poetry though automates and manages all of that for you. With poetry init you can initialise a venv for your project and with Poetry shell you can activate it. Installing and removing a dependency like Django is as easy as poetry add Django and respectively poetry remove Django. All this without having to activate the virtual environment of your Python project first since poetry keeps track of its location when it is first created.\nSeparation between runtime and development dependencies #You can now separate between runtime dependencies and development dependencies in your Python project.\nFor example, you might want your project to use Django and Gunicorn for creating and running a web app but also flake8 and pytest for linting and testing it. Poetry makes this possible just by running the following commands:\npoetry add django gunicorn poetry add flake8 pytest --group test The result will be a pyproject.toml file which will contain the following lines:\n[tool.poetry.dependencies] python = \u0026#34;^3.9\u0026#34; Django = \u0026#34;^3.1.6\u0026#34; gunicorn = \u0026#34;^20.0.4\u0026#34; [tool.poetry.dev-dependencies] flake8 = \u0026#34;^3.8.4\u0026#34; pytest = \u0026#34;^6.2.2\u0026#34; If at some point your project needs to be deployed on a production environment or inside a lightweight Docker container then you can use poetry install --no-dev to install only its runtime dependencies.\nVersion constraints #Recently pip reimplemented the ability to define constraints that control the version of your dependencies that are allowed to be installed, by using \u0026hellip;an additional constraints.txt file.\nPoetry follows a more traditional (saner?) approach by using similar syntax (tildes, carets, wildcards) that you would find in npm or composer. Anyone coming from these backgrounds should be comfortable in defining dependencies in Poetry and its documentation provides many examples. In my example above, the caret (^) symbol would allow updates to versions of Django from 3.1.6, up to but not including version 4.\nSub dependencies and conflict resolution #Poetry keeps track of the dependency tree from your defined dependencies and handles any conflicts. Similarly to pip, if two or more dependencies require the same sub-dependency, then Poetry will attempt to use the version of the sub-dependency that is compatible with all them.\nPoetry goes a step further than pip though. If a sub-dependency is not required by any of the main dependencies it will also be removed, something which unfortunately has not been possible so far with pip. No more orphaned dependencies on your environment!\nAll this is possible because of\u0026hellip;\nLockfiles #In addition to the standard pyproject.toml that contains the main dependencies of your project, poetry generates a lockfile poetry.lock that makes all of the above possible. Effectively it resolves and \u0026lsquo;freezes\u0026rsquo; the dependency tree of your project to its last working state. In fact, it is even recommended to commit this lockfile in any version control (like Git) so that anyone who uses your application will run it on the exact same dependencies specified in it. The generation of this file is automatically handled by poetry and you will never have to edit it manually.\nFor example, applications that require Django would have a snippet like the one below in their lockfile that details exactly what is needed to be installed along with it:\n[[package]] name = \u0026#34;django\u0026#34; version = \u0026#34;3.1.6\u0026#34; description = \u0026#34;A high-level Python Web framework that encourages rapid development and clean, pragmatic design.\u0026#34; category = \u0026#34;main\u0026#34; optional = false python-versions = )=3.6\u0026#34; [package.dependencies] asgiref = )=3.2.10,\u0026lt;4\u0026#34; pytz = \u0026#34;\\*\u0026#34; sqlparse = )=0.2.2\u0026#34; Use Poetry with different Python versions #If you have multiple installed versions of Python on your machine, you can point Poetry to the one that you want to use with your project with:\npoetry env use /full/path/to/python Export dependencies to requirements.txt files #Finally, if for some reason you are required to use a requirements.txt file for your project (maybe using a minimal Python Docker image that contains only pip), poetry can export dependencies in that format with one command:\npoetry export -f requirements.txt --output requirements.txt Bonus items # Poetry by default stores a generated virtual environment in a local directory in your home folder, outside the project root. My personal preference is to have it in a.venvfolder at the root of my project. Poetry gives you this option which can be also configured as an environment variable. I usually add this to my .bashrc: export POETRY_VIRTUALENVS_IN_PROJECT=1 PyCharm has added support for Poetry since version 2021.3.\nVisual Studio Code has added built-in support for Poetry since its April 2021 Release.\nFinal thoughts #Try it yourself! If you are creating something with Python and want to try a more opinionated and efficient way to manage your project\u0026rsquo;s dependencies, try Poetry!\ncurl -sSL https://install.python-poetry.org | python3 - mkdir my_project cd my_project poetry init ","date":"14 October 2022","permalink":"https://www.angelospanag.me/posts/dependency-management-in-python-using-poetry/","section":"Posts","summary":"A modern way to do dependency management in your Python projects","title":"Dependency management in Python using Poetry"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/tags/virtual-environments/","section":"Tags","summary":"","title":"Virtual Environments"},{"content":"","date":null,"permalink":"https://www.angelospanag.me/categories/","section":"Categories","summary":"","title":"Categories"}]