Search This Blog

Thursday, June 22, 2023

FastAPI and Swagger2

 I recently started working on a web project with Python backend. The hosting was on Google Cloud. 

 We built the POC version of the Python backend using the most popular Python3 framework FastAPI. We deployed it on Cloud Run (public) with Docker image and everything worked as expected. 

 We then moved on to making the Cloud Run private and adding an API gateway infront of it. We saved the http://localhost:8000/openapi.json and ran the gcloud cli to create an API gateway and it failed!

 Looking at the documentation https://cloud.google.com/api-gateway/docs/openapi-overview we realized, Google, for some reason, still only supports OpenAPI 2.0 (Swagger 2.0). Working on AWS for quite some time, I had assumed OpenAPI 3 would be supported.

 We tried figuring out a way to configure FastAPI to generate Swagger 2 spec. But FastAPI only supports OpenAPI 3.x. Many have suggested downloading the openapi.json, passing it through some convertor and then some manual intervention to convert it to Swagger 2. 

 Since I wanted to use CI/CD, I didn't want to deal with manual conversion every time developers made some change. So, I referred the FastAPI OpenAPI related code and came up with Package version

Requirements

Python 3.8+
FastAPI 0.79.0+

Installation

$ pip install fastapi_swagger2

Example

from typing import Union
from fastapi import FastAPI
from fastapi_swagger2 import FastAPISwagger2

app = FastAPI()
FastAPISwagger2(app)


@app.get("/")
def read_root():
    return {"Hello": "World"}


@app.get("/items/{item_id}")
def read_item(item_id: int, q: Union[str, None] = None):
    return {"item_id": item_id, "q": q}


This adds following endpoints:
http://localhost:8000/swagger2.json
http://localhost:8000/swagger2/docs
http://localhost:8000/swagger2/redoc


Generate spec for CI/CD

import os

import yaml

from app.main import app

URL = os.environ["CLOUD_RUN_URL"]

app.servers.append(URL)

spec = app.swagger2()
spec['x-google-backend'] = {'address': URL}

print(yaml.dump(spec))


Monday, April 12, 2021

Run GUI applications inside Multipass (Ubuntu) on macOS

Multipass is a nice docker like vm platform from Ubuntu. It can configure the vm with cloud-init so you can prototype your cloud launches locally for free. 

E.g. Launch an instance (by default you get the current Ubuntu LTS)

$ multipass launch --name foo

You can read more about multipass in the docs.

Primarly it does seem like just another alternative to docker. But then I found one nice little difference, not a resource hog unlike docker and you don't run as root.

Currently to run a GUI application which you just want to try without having to clean up the system, most common approach is create a vm with VirtualBox, VMware Fusion or Parallels. But it can be a time consuming task as it installs everything from scratch.

Getting GUI applications to work on docker is possible configuration. But then, atleast on Mac, Docker is a resource hog.

With multipass I was able to get IntelliJ IDEA with following:

Prerequisites:

  • Install Multipass
  • Install XQuartz. Also configure XAuthLocation.
    echo "XAuthLocation /opt/X11/bin/xauth" >> ~/.ssh/config

Launch a vm with current LTS ubuntu with 4 cores CPU and 4 GB memory

$ multipass launch -n idea -c 4 -m 4G

$ multipass shell idea

$ sudo apt update

$ sudo apt install -y libxrender-dev libxtst6 libxi6 fontconfig

$ cd /tmp

$ curl -L -O https://download.jetbrains.com/idea/ideaIC-2021.1.tar.gz

$ sudo tar -xzf ideaIC-2021.1.tar.gz -C /opt

$ echo 'PATH=/opt/idea-IC-211.6693.111/bin:$PATH' >> ~/.bashrc

$ sudo apt install -y openjdk-11-jdk

You could automate the above with cloud-init. (Currently as of this writing multipass only supports yaml configs)

Now time for the magic part (on the host):

multipass shell internally is just doing a SSH

$ sudo cp '/var/root/Library/Application Support/multipassd/ssh-keys/id_rsa' ~/.ssh/multipass_id_rsa

Save this script to your PATH

$ cat ~/bin/multipass_x_ssh

 

#!/bin/bash

IP=$(multipass info $1 | grep IPv4 | awk '{ print $2 }')

ssh -X -i ~/.ssh/multipass_id_rsa ubuntu@$IP

Now when I want to run idea, all I have to do is:

$ multipass_x_ssh idea

$ idea.sh

Saturday, February 27, 2021

pyenv install error on MacOS Big Sur - sendfile invalid

There seems to be some issue going on with pyenv and MacOS Big Sur.


./Modules/posixmodule.c:8320:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]

        ret = sendfile(in, out, offset, &sbytes, &sf, flags);


After referring multiple posts, this is what worked for me.

brew install zlib bzip2

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" pyenv install 3.5.3 --patch < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

Wednesday, February 10, 2021

Vmware Fusion 12 - Mac OS Big Sur - Could not connect Network

Was unable to get the network to work after a restart.

| vmx| I005: VNET: 'ethernet0' enable link state propagation, lsp.state = 5

| vmx| I005: DictionaryLoad: Cannot open file "/Library/Preferences/VMware Fusion/config": No such file or directory.

| vmx| I005: VNET: MACVNetMacosGetRealAdapterType: network type for adapter 0: 8

| vmx| I005: VNET: MACVNetMacosGetVnetProperties: vnet properties: vnet=vmnet8, nat=yes, dhcp=yes (ignored), subnet=172.16.82.0, mask=255.255.255.0, firstAddr=172.16.82.1, lastAddr=172.16.82.127, isIPv6=no, IPv6Prefix=fd15:4ba5:5a2b:1008::, IPv6PrefixLen=64

| vmx| I005: VNET: MACVNetPortVirtApiStartInterface: Waiting on semaphore for adapter: 0

| host-1216256| I005: VNET: MACVNetPortVirtApiStartHandler: starting interface for adapter: 0, status: 1009

| host-1216256| W003: VNET: MACVNetPortVirtApiStartHandler: unable to create virtual intrface for device: 0, status: 1009

| vmx| I005: VNET: Semaphore signalled for adapter: 0, timeoutMs=5000, waitMs=3

| vmx| W003: VNET: MACVNetPortVirtApiStartInterface: Failed to create interface for adapter: 0, handlerStatus: 1009

| vmx| I005: VNET: MACVNetPort_Connect: Ethernet0: can't start virtual interface

| vmx| I005: Msg_Post: Error

| vmx| I005: [msg.vnet.connectvnet] Could not connect 'Ethernet0' to virtual network '/dev/vmnet8'. More information can be found in the vmware.log file.

| vmx| I005: [msg.device.badconnect] Failed to connect virtual device 'Ethernet0'.

| vmx| I005: ----------------------------------------


Here's what worked for me:

  • Click on "Apple" logo in menu bar
  • Click "System Preferences"
  • Click "Sharing"
  • Disable "Internet Sharing"
  • Restart the VM.

 


Saturday, June 20, 2020

MacOS pyenv python 3.8 tkinter install


brew install pyenv

brew install tcl-tk

echo 'export PATH="$(brew --prefix tcl-tk)/bin:$PATH"' >> ~/.zshrc
export LDFLAGS="-L$(brew --prefix tcl-tk)/lib"
export CPPFLAGS="-I$(brew --prefix tcl-tk)/include"
export PKG_CONFIG_PATH="$(brew --prefix tcl-tk)/lib/pkgconfig"


PYTHON_CONFIGURE_OPTS="--with-tcltk-includes='-I$(brew --prefix tcl-tk)/include' --with-tcltk-libs='-L$(brew --prefix tcl-tk)/lib -ltcl8.6 -ltk8.6'" CFLAGS="-I$(brew --prefix tcl-tk)/include" pyenv install 3.8.3

Monday, September 10, 2018

Duply backup on MacOS

Download duplicity from http://duplicity.nongnu.org/

untar

cd duplicity-0.7.18

mkvirtualenv duply_backup

brew install librsync
brew install gpg
pip install fasteners

python setup install

Download duply from https://sourceforge.net/projects/ftplicity/files/duply%20%28simple%20duplicity%29/

untar

cd duply_2.1

workon duply_backup

./duply -V
  duply version 2.1
  (http://duply.net)

  Using installed duplicity version 0.7.18, python 2.7.14 (/Users/vkanwade/.virtualenvs/duply_backup/bin/python2), gpg 2.2.10 (Home: /Users/vkanwade/.gnupg), awk 'awk version 20070501', grep 'grep (BSD grep) 2.5.1-FreeBSD', bash '3.2.57(1)-release (x86_64-apple-darwin17)'.


export DUPL_PYTHON_BIN=~/.virtualenvs/duply_backup/bin/python
export PATH="$PATH:/opt/duply_2.1/"

duply myprofile create
vim ~/.duply/myprofile/conf

#GPG_KEY='_KEY_ID_'
GPG_PW='<some random pwd generated with pwgen. store it safe, required for restore>'

TARGET='s3://< s3_prefix>< s3_region >.amazonaws.com/< s3_bucket >/duply/myprofile'
# s3_prefix: eu-central-1 's3.', else 's3-'

export AWS_ACCESS_KEY_ID='< aws_s3_backup_id >'
export AWS_SECRET_ACCESS_KEY='< aws_s3_backup_secret >'
DUPL_PARAMS="$DUPL_PARAMS --allow-source-mismatch --s3-use-new-style --s3-european-buckets "

SOURCE='/'

MAX_AGE=1M

MAX_FULLBKP_AGE=1M
DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "

ARCH_DIR=~/duply-cache

NOTE: just wanted to dump the info. formatting will come.

Tuesday, October 25, 2016

MacOS - Operation not permitted - Can't upgrade package 'six'

For most of the packages, python-dateutil is the one which causes all the issue. So just upgrade it an all other installations should be fine.

sudo -H pip install --upgrade python-dateutil --ignore-installed six