Right way to manage a high traffic connection application

Carlos

Introduction

I am working on a GPS Listener, this is a service build on twisted python, this app receive at least 100 connections from gps devices, and it is working without issues, each GPS send data each 5 seconds, containing positions. ( the next week must be at least 200 gps devices connected )

Database

I am using a unique postgresql connection, this connection is shared between all gps devices connected for save and store information, postgresql is using pgbouncer as pooler

Server

I am using a small pc as server, and I need to find a way to have a high availability application with out loosing data

Problem

According with my high traffic on my app, I am having issues with memory data after 30 minutes start to appear as no saved, however queries are being executed on postgres ( I have checked that on last activity )

Fake Solution

I have amke a script that restart my app, postgres ang pgbouncer, however this is a wrong solution, because each time that I restart my app, gps get disconnected, and must to reconnected again

Posible Solution

I am thinking on a high availability solution based on a Data Layer, where each time when database have to be restarted or something happened, a txt file store data from gps devices.

For get it, I am thing on a no unique connection, I am thinking on a simple connection each time one data must be saved, and then test database, like a pooler, and then if database connection is wrong, the txt file store it, until database is ok again, and the other process read txt file and send info to database

Question

Since I am thinking on a app data pooler and a single connection each time when this data must be saved for try to no lost data, I want to know

Is ok making single connection each time that data is saved for this kind of app, knowing that connections will be done more than 100 times each 5 seconds?

As I said, my question is too simple, which one is the right way on working with db connections on a high traffic app? single connections per query or shared unique connection for all app.

The reason on looking this single question, is looking for the right way on working with db connections considering memory resources.

I am not looking for solve postgresql issues or performance, just to know the right way on working with this kind of applications. And that is the reason on give as much of possible about my application

Note One more thing,I have seen one vote to close this question, that is related to no clear question, when the question is titled with the word "question" and was marked on italic, now I have marked as gray for notice people that dont read the word "question"

Thanks a lot

jjanes

Databases do not just lose data willy-nilly. Not losing data is pretty much number one in their job description. If it seems to be losing data, you must be misusing transactions in your application. Figure out what you are doing wrong and fix it.

Making and breaking a connection between your app and pgbouncer for each transaction is not good for performance, but is not terrible either; and if that is what helps you fix your transaction boundaries then do that.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

Connection management when using kafka producer in high traffic environment

Right way to manage components functionality in React

Best way to manage database connection for a Java servlet

Azure Application Insights - Not recording all requests on high traffic situations

What is the best way to manage application screens in SWT?

SSH Connection with keys (no password) follows a One-way traffic?

Right way to manage Arrays with intersecting types and narrow them back down

NodeJS + ExpressJS proper way to manage connection to mysql DB

High traffic hosting provider

Right way to transfer a CSV file to a BI application?

How to Deviceready in right way in Ionic application?

Is this the right way to add custom fields in Application Insight?

What is the right way to monitor application memory usage?

Right way to define Procfile for Nuxt + Php application

Is there any way configure Application Gateway for Least Connection?

Best way to manage the ActiveWorkbook in a VSTO Excel Addin application

Can Stargate handle high traffic?

The way to solve cpu load too high of Java application

What is the right way to manage synthetic events using hooks and update the state from event handlers?

is .htaccess + mod_rewrite the right way to manage the permission for epub-files (epub-shop)?

Is there any way to launch my application right away without 'Ok Google"?

Killing an application, is Android going the right way in this process.?

What is the right way to use Cassandra driver from a web application

Right way to capture push notifications when the application is open in Android

Is it the right way to integrate REST Web Service with Activemq in Spring Application

How to make setup.py for standalone python application in a right way?

Proper way to manage test/prod connection strings in .NET web.config file

How to use C3P0 in a spring boot hibernate multi tenant application to manage connection pool?

What is the best way in WPF to implement traffic light colours for different status's in the application?