jblu/Inex
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects 1 Releases 8 Wiki Activity

Compare commits

base: jblu:main
Branches Tags
jblu:main
jblu:middleofrebase
jblu:v.1.1.4
jblu:v.1.1.3
jblu:v.1.1.0
jblu:v.1.0.0
jblu:v.0.0.4
jblu:v.0.0.3
jblu:v.0.0.2
jblu:Basic
..
compare: jblu:v.1.0.0
Branches Tags
jblu:main
jblu:middleofrebase
jblu:v.1.1.4
jblu:v.1.1.3
jblu:v.1.1.0
jblu:v.1.0.0
jblu:v.0.0.4
jblu:v.0.0.3
jblu:v.0.0.2
jblu:Basic

No commits in common. "main" and "v.1.0.0" have entirely different histories.
main ... v.1.0.0

10 changed files with 153 additions and 375 deletions
Show Stats Expand all files Collapse all files

4
.gitea/workflows/build.yaml
View File

@ -38,6 +38,4 @@ jobs:
- run: pyinstaller --noconfirm --onefile --console ${{ gitea.workspace }}/inex.py - run: pyinstaller --noconfirm --onefile --console ${{ gitea.workspace }}/inex.py
- uses: softprops/action-gh-release@v2 - uses: softprops/action-gh-release@v2
with: with:
files: | files: ${{ gitea.workspace }}/dist/inex.exe
${{ gitea.workspace }}/dist/inex.exe
${{ gitea.workspace }}/config.toml.example

121
README.md
View File

@ -1,93 +1,34 @@
# Inex # Inex
### Information Exchange - Ingest ARM data into EFC Information Exchange - Pull data from EFT ARM then dump it to a json file
This application is designed to pull data from an Microsoft SQL database, reformat it, and then send that data via the web to the Event Fusion Center. It's designed to be performant. The embedded query doesn't return unneccesary data and data is written to memory before being pushed into EFC. Initially you may have to pull a large amount of days so the system is caught up but the sweet spot would be to pull a set of data every day.
## Build from source # Data Map:
You will need to install git and clone the repository: ## tbl_ProtocolCommands
- ProtocolCommandID
`git clone https://git.jonb.io/jblu/Inex.git` - Time_stamp
- RemoteIP
Alternitively you can download the files from [the releases page](https://git.jonb.io/jblu/Inex/releases). - RemotePort
- LocalIP
Then follow OS specific instructions. These are taken directly from the steps used in automation to build the executables found in the releases page. - LocalPort
##### Windows - Protocol
- SiteName
You can find the requirements.txt in the repository. - Command
- CommandParameters
`python -m pip install -r requirements.txt` - FileName
- VirtualFolderName
`pyinstaller --noconfirm --onefile --console path/to/inex.py` - PhysicalFolderName
- IsInternal
##### Linux - FileSize
Update Linux Distro. This is Ubuntu specific. Please check the package management tool in your specific version. - TransferTime
- BytesTransferred
`apt-get update` - ResultID
- TransactionID
You need unixodbc or else pyodbc cannot install. - Description
- Actor
`apt-get install unixodbc -y` ## tbl_Transactions
- TransactionID
You can find the requirements.txt in the repository. - ParentTransactionID
- TransactionObject
`pip install -r requirements.txt` - TransactionGUID
- NodeName
`pyinstaller --noconfirm --onefile --console path/to/inex.py` - time_stamp
## Setup
You will need a *config.toml* file in the same directory where *inex.py* or inex executable is. It's recommended to rename the existing config.toml.example file to save sometime and avoid omitting any data.
> If the application doesn't find the *config.toml* at the time of execution, it will not continue.
#### Config.toml
| Table | Purpose |
|-|-|
| fortraPlatform | Fortra Specific data |
| database | MSSQL Configuration |
| immutables | Data that must be included but is not provided by the source database |
| output | If and where to write files|
| logging | Set if, where and level of logging |
The following settings are not obvious as to how they affect the application.
> Note the #comments
```
[fortraPlatform]
selectedPlatform = "dev" # This will modify which environment the data is pushed to. The tenant_id and secret must be manually modified.
[database]
overrideEmbeddedquery = true # Choose if embedded query should be overridden.
daysTopull = 30 # This setting is only related to the embedded query. Please note this will not affect query provided in config.toml
driver = "ODBC Driver 18 for SQL Server" # Select which windows driver should be used. This one is recommended.
[output]
pushToplatform = false # if true, send data to fortraPlatform setting to false is useful in dry-runs.
dumpTojson = true # if true, write data to json file
token = "./.token" # Recommended to leave as-is
```
## Usage
### Windows
##### Run inex.py
1. Download [source](https://git.jonb.io/jblu/Inex/releases)
2. `cd C:\path\to\Inex`
3. `python inex.py`
OR
##### Run inex.exe
1. Download [source](https://git.jonb.io/jblu/Inex/releases)
2. `cd C:\path\to\Inex.exe`
3. CMD `.\inex.exe`
### Linux
##### Run inex.py
1. Download [source](https://git.jonb.io/jblu/Inex/releases)
2. `cd /path/to/Inex`
3. `python ./inex.py`
OR
1. Download [source](https://git.jonb.io/jblu/Inex/releases)
2. `cd /path/to/Inex`
3. RUN `./inex`

59
config.toml.example
View File

@ -1,33 +1,6 @@
[fortraPlatform] [fortraPlatform]
selectedPlatform = "dev" selectedPlatform = "dev"
[output]
pushToplatform = false
dumpTojson = true
filename ="./data.json"
token = "./.token"
[logging]
useLog = true
logLevel = "debug"
logPath = "./inex.log"
[database]
overrideEmbeddedquery = false
daysTopull = 20
driver = "ODBC Driver 18 for SQL Server"
server = "192.168.x.x"
database = "EFTDB"
user = "a"
password = "a"
query = """DECLARE @stopTime DATETIME2
SET @stopTime=DATEADD(DAY, -30, GETDATE())
SELECT p.ProtocolCommandID, t.Time_stamp, p.RemoteIP, p.RemotePort, p.LocalIP, p.LocalPort, p.Protocol, p.SiteName, p.Command, p.FileName, p.PhysicalFolderName, p.VirtualFolderName, p.FileSize, p.TransferTime, p.BytesTransferred, p.Description, p.ResultID, t.TransactionID, p.Actor, t.TransactionObject, t.NodeName, t.TransactionGUID, a.Protocol user_type
FROM tbl_Transactions t
Full JOIN tbl_ProtocolCommands p ON(t.TransactionID=p.TransactionID)
Full JOIN tbl_Authentications a ON(t.TransactionID=a.TransactionID)
WHERE p.Time_stamp>@stopTime AND p.Command IS NOT NULL"""
[fortraPlatform.dev] [fortraPlatform.dev]
idp = "https://foundation.foundation-dev.cloudops.fortradev.com/idp/realms/products/protocol/openid-connect/token" idp = "https://foundation.foundation-dev.cloudops.fortradev.com/idp/realms/products/protocol/openid-connect/token"
efc_url = "https://efc.efc-dev.cloudops.fortradev.com" efc_url = "https://efc.efc-dev.cloudops.fortradev.com"
@ -49,6 +22,34 @@ tenant_id = ""
client_id = "eft-event-generator-confidential" client_id = "eft-event-generator-confidential"
secret = "" secret = ""
[database]
overrideEmbeddedquery = false
driver = "ODBC Driver 18 for SQL Server"
server = "192.168.x.x"
database = "EFTDB"
user = "a"
password = "a"
query = """DECLARE @stopTime DATETIME2
SET @stopTime=DATEADD(DAY, -30, GETDATE())
SELECT p.ProtocolCommandID, t.Time_stamp, p.RemoteIP, p.RemotePort, p.LocalIP, p.LocalPort, p.Protocol, p.SiteName, p.Command, p.FileName, p.VirtualFolderName, p.FileSize, p.TransferTime, p.BytesTransferred, p.Description, p.ResultID, t.TransactionID, p.Actor, t.TransactionObject, t.NodeName, t.TransactionGUID, a.Protocol user_type
FROM tbl_Transactions t
Full JOIN tbl_ProtocolCommands p ON(t.TransactionID=p.TransactionID)
Full JOIN tbl_Authentications a ON(t.TransactionID=a.TransactionID)
WHERE p.Time_stamp>@stopTime AND p.Command IS NOT NULL"""
[immutables] [immutables]
product_name = "GlobalScape EFT" prd_instance_id = 1
prd_ext_tenant_name = "GlobalScape EFT" product_guid = "asdf"
product_name = "EFT"
product_version ="8.1.0.9"
[output]
pushToplatform = true
dumpTojson = true
filename ="./data.json"
token = "./.token"
[logging]
use_log = true
logLevel = "debug"
logPath = "./inex.log"

32
inex.py
View File

@ -13,10 +13,7 @@ import inexSqlquery
class Inex: class Inex:
def __init__(self): def __init__(self):
"""Initilize config, calls functions from inexConnect.py, inexLogging.py """Initilize config, calls functions from inex-connect.py and inex-logging.py"""
inexDataModel.py, inexDataProcessing.py, inexEncoder.py and inexSqlquery.py
Main logic of the program. Requires a config.toml in the same directory it's
being run from."""
# assign libraries # assign libraries
self.db = pyodbc self.db = pyodbc
self.il = logging self.il = logging
@ -28,7 +25,6 @@ class Inex:
self.e = inexEncoder.Encoder self.e = inexEncoder.Encoder
self.sq = inexSqlquery self.sq = inexSqlquery
# Check if local config file exists.
if self.os.path.exists('./config.toml'): if self.os.path.exists('./config.toml'):
config_file_path = './config.toml' config_file_path = './config.toml'
with open(config_file_path, 'rb') as c: with open(config_file_path, 'rb') as c:
@ -47,19 +43,17 @@ class Inex:
self.useLog = self.config["logging"]["useLog"] self.useLog = self.config["logging"]["useLog"]
self.logPath = self.config["logging"]["logPath"] self.logPath = self.config["logging"]["logPath"]
self.logLevel = self.config["logging"]["logLevel"] self.logLevel = self.config["logging"]["logLevel"]
self.prdExttenantname = self.config["immutables"]["prd_ext_tenant_name"] self.prdInstanceID = self.config["immutables"]["prd_instance_id"]
self.productGUID = self.config["immutables"]["product_guid"]
self.productName = self.config["immutables"]["product_name"] self.productName = self.config["immutables"]["product_name"]
self.productVersion = self.config["immutables"]["product_version"]
self.tokenFilepath = self.config["output"]["token"] self.tokenFilepath = self.config["output"]["token"]
self.selectedPlatform = self.config["fortraPlatform"]["selectedPlatform"] self.selectedPlatform = self.config["fortraPlatform"]["selectedPlatform"]
self.writeJsonfile = self.config["output"]["dumpTojson"] self.writeJsonfile = self.config["output"]["dumpTojson"]
self.pushToplatform = self.config["output"]["pushToplatform"] self.pushToplatform = self.config["output"]["pushToplatform"]
self.queryOverride = self.config["database"]["overrideEmbeddedquery"] self.queryOverride = self.config["database"]["overrideEmbeddedquery"]
self.queryDaystopull = self.config["database"]["daysTopull"] except:
except Exception as e: print("No config.toml. Please use example file and configure appropriately")
print("No config.toml or possibly missing settings in the file. Please use config.toml.example file and configure appropriately")
self.il.error(e)
print(e)
exit(1) exit(1)
if "dev" in self.selectedPlatform.lower(): if "dev" in self.selectedPlatform.lower():
@ -75,20 +69,18 @@ class Inex:
# create the connection to the database # create the connection to the database
self.cursor = self.ic.inexSql.connectDatabase(self, self.db, self.dbDriver, self.dbServer, self.dbDatabase, self.dbUser, self.dbPassword) self.cursor = self.ic.inexSql.connectDatabase(self, self.db, self.dbDriver, self.dbServer, self.dbDatabase, self.dbUser, self.dbPassword)
# Query the database
self.data = self.ic.inexSql.databaseQuery(self, self.cursor, self.sq.sqlQuerymodel.queryData(self.queryOverride,self.dbQuery, self.queryDaystopull))
# Modify the data to meet EFC requirements self.data = self.ic.inexSql.databaseQuery(self, self.cursor, self.sq.sqlQuerymodel.queryData(self.queryOverride,self.dbQuery))
self.modifiedData = processData(self.data, dataTemplate, prd_ext_tenant_name=self.prdExttenantname,product_name=self.productName,\
prd_ext_tenant_id=self.platformConfig["tenant_id"]) self.modifiedData = processData(self.data, dataTemplate, prd_instance_id=self.prdInstanceID,\
product_guid=self.productGUID,product_name=self.productName,product_version=self.productVersion)
# Push data to EFC. Check for local Auth token -> Authenticate if needed -> push data
if self.pushToplatform: if self.pushToplatform:
inexConnect.fortraEFC.__init__(self) inexConnect.fortraEFC.pushPayload(self)
# TODO: move this to its own function
if self.useLog: if self.useLog:
self.il.warning(f"Writing to '{self.outputFile}'.") self.il.warning(f"Writing to '{self.outputFile}'.")
# Write data to json
if self.writeJsonfile: if self.writeJsonfile:
with open(self.outputFile, "w") as f: with open(self.outputFile, "w") as f:
self.j.dump(self.modifiedData, f, indent = 2, cls=self.e) self.j.dump(self.modifiedData, f, indent = 2, cls=self.e)

59
inexConnect.py
View File

@ -20,7 +20,6 @@ class inexSql:
return cursor return cursor
def databaseQuery(self, cursor, query, args=()): def databaseQuery(self, cursor, query, args=()):
"""Use the database connection to send a query."""
if self.useLog: if self.useLog:
self.il.debug(f"Query:") self.il.debug(f"Query:")
self.il.debug(query) self.il.debug(query)
@ -44,54 +43,32 @@ class inexSql:
return r return r
class fortraEFC: class fortraEFC:
"""Class to connect to fortra EFC. It will authenticate and push rest payloads.
Writes a .token file to the same directory script was run in."""
def __init__(self):
"""This is the logic for how authentication is handled"""
# Check if .token file is present
if fortraEFC.readToken(self) == 1:
# Get fresh token. First run.
fortraEFC.getToken(self)
fortraEFC.writeToken(self)
# Push data with token
self.pushPayloadresponse = fortraEFC.pushPayload(self)
if self.pushPayloadresponse == 401:
fortraEFC.getToken(self)
fortraEFC.writeToken(self)
fortraEFC.pushPayload(self)
def readToken(self):
"""Looks locally for a .token file. Returns a numeral code
for logic in the init method."""
if self.os.path.exists(self.tokenFilepath):
with open(self.tokenFilepath, 'rb') as t:
self.tokenData = self.j.load(t)
self.il.debug(f'readToken {self.tokenData["access_token"]}')
return 0
else:
return 1
def getToken(self): def getToken(self):
"""Gets a token from fortra idp."""
self.tokenData = self.r.post(self.platformConfig["idp"], data={"grant_type":"client_credentials",\ self.tokenData = self.r.post(self.platformConfig["idp"], data={"grant_type":"client_credentials",\
"client_id": self.platformConfig["client_id"],\ "client_id": self.platformConfig["client_id"],\
"client_secret": self.platformConfig["secret"],}) "client_secret": self.platformConfig["secret"],})
self.tokenData = self.tokenData.json()
self.il.debug(f'getToken {self.tokenData["access_token"]}')
def writeToken(self): def writeToken(self):
"""Writes a token to a local file named '.token'."""
fortraEFC.getToken(self) fortraEFC.getToken(self)
with open(self.tokenFilepath, "w") as f: with open(self.tokenFilepath, "w") as f:
self.j.dump(self.tokenData, f, indent = 2) self.j.dump(self.tokenData.json(), f, indent = 2)
self.il.debug(f'writeToken {self.tokenData["access_token"]}')
def readToken(self):
if self.os.path.exists(self.tokenFilepath):
with open(self.tokenFilepath, 'rb') as t:
self.tokenData = self.j.load(t)
# print(self.tokenData["access_token"])
else:
fortraEFC.writeToken(self)
def pushPayload(self): def pushPayload(self):
"""Sends data to fortra EFC. Requires a token from the idp.""" fortraEFC.readToken(self)
self.il.debug(f'pushPayload {self.tokenData["access_token"]}') try:
url = f'{self.platformConfig["efc_url"]}/api/v1/unity/data/{self.platformConfig["tenant_id"]}/machine_event' url = f'{self.platformConfig["efc_url"]}/api/v1/unity/data/{self.platformConfig["tenant_id"]}/machine_event'
pushPayloadResponse = self.r.post(url, headers={'Authorization': f'Bearer {self.tokenData["access_token"]}'},\ pushPayloadResponse = self.r.post(url, headers={'Authorization': f'bearer {self.tokenData["access_token"]}'},\
data=self.j.dumps(self.modifiedData, cls=self.e)) json=self.j.dumps(self.modifiedData,indent = 2, cls=self.e))
self.il.debug(pushPayloadResponse.status_code)
self.il.debug(pushPayloadResponse.text)
return pushPayloadResponse.status_code return pushPayloadResponse.status_code
except self.r.exceptions.HTTPError as errh:
print ("Http Error:",errh)
if "401" in errh:
fortraEFC.writeToken(self)
fortraEFC.pushPayload(self)

114
inexDataModel.py
View File

@ -1,10 +1,6 @@
def dataTemplate(transactionType,**kwargs): def dataTemplate(transactionType,**kwargs):
"""Created templates for use. This function forms json data into an uploadDownload = {
appropriate model for EFC. It returnes the appropriate template based "bytes" : kwargs.get('bytes_out'),
on the transaction type passed into the function. The logic to process
this is at the bottom of the function."""
upload = {
"bytes" : kwargs.get('bytes'),
"dst_endpoint": { "dst_endpoint": {
"port": kwargs.get('dst_endpoint_port'), "port": kwargs.get('dst_endpoint_port'),
"ip": kwargs.get('dst_endpoint_ip'), "ip": kwargs.get('dst_endpoint_ip'),
@ -12,17 +8,16 @@ def dataTemplate(transactionType,**kwargs):
}, },
"duration": kwargs.get('duration'), "duration": kwargs.get('duration'),
"file": { "file": {
"created_time": kwargs.get('file_created_time'), "created_time": kwargs.get('time'),
"uid": kwargs.get('file_uid'),
"size": kwargs.get('file_size'), "size": kwargs.get('file_size'),
"name": kwargs.get('file_name'), "name": kwargs.get('file_name'),
"path": kwargs.get('file_path') "path": kwargs.get('file_path')
}, },
"guid": kwargs.get('file_uid'), "guid": kwargs.get('guid'),
"node_name": kwargs.get('node_name'), "node_name": kwargs.get('node_name'),
"prd_ext_tenant_id": kwargs.get('prd_ext_tenant_id'), "prd_ext_tenant_id": kwargs.get('tenant'),
"product_name": kwargs.get('product_name'), "product_name": "GlobalScape EFT",
"prd_ext_tenant_name": kwargs.get('prd_ext_tenant_name'), "prd_ext_tenant_name": "GlobalScape EFT",
"classifications": [{ "classifications": [{
"ref_id": f"globalscape:{kwargs.get('guid')}", "ref_id": f"globalscape:{kwargs.get('guid')}",
"time": kwargs.get('time'), "time": kwargs.get('time'),
@ -36,61 +31,15 @@ def dataTemplate(transactionType,**kwargs):
"ip": kwargs.get('src_endpoint_ip'), "ip": kwargs.get('src_endpoint_ip'),
"type": kwargs.get('src_endpoint_type') "type": kwargs.get('src_endpoint_type')
}, },
"tenant": kwargs.get('prd_ext_tenant_id'), "tenant": kwargs.get('tenant'),
"tenant_name":"GlobalScape", "tenant_name":"GlobalScape",
"time": kwargs.get('time'), "time": kwargs.get('time'),
"status_code": kwargs.get('status_code'), "status_code": kwargs.get('status_code'),
"status_detail": kwargs.get('status_detail'), "status_detail": kwargs.get('description'),
"user": { "user": {
"home_directory": kwargs.get('user_home_directory'), "home_directory": kwargs.get('user_home_directory'),
"uuid": kwargs.get('guid'), "uuid": kwargs.get('guid'),
"uid": kwargs.get('user_uid'), "uid": kwargs.get('uid'),
"type": kwargs.get('user_type'),
"name": kwargs.get('user_name')
},
"utype": kwargs.get('utype')
}
download = {
"bytes" : kwargs.get('bytes'),
"dst_endpoint": {
"port": kwargs.get('dst_endpoint_port'),
"ip": kwargs.get('dst_endpoint_ip'),
"type": kwargs.get('dst_endpoint_type')
},
"duration": kwargs.get('duration'),
"file": {
"uid": kwargs.get('file_uid'),
"size": kwargs.get('file_size'),
"name": kwargs.get('file_name'),
"path": kwargs.get('file_path')
},
"guid": kwargs.get('file_uid'),
"node_name": kwargs.get('node_name'),
"prd_ext_tenant_id": kwargs.get('prd_ext_tenant_id'),
"product_name": kwargs.get('product_name'),
"prd_ext_tenant_name": kwargs.get('prd_ext_tenant_name'),
"classifications": [{
"ref_id": f"globalscape:{kwargs.get('guid')}",
"time": kwargs.get('time'),
}],
"session": {
"created_time": kwargs.get('time'),
"uid": kwargs.get('session_uid')
},
"src_endpoint": {
"port": kwargs.get('src_endpoint_port'),
"ip": kwargs.get('src_endpoint_ip'),
"type": kwargs.get('src_endpoint_type')
},
"tenant": kwargs.get('prd_ext_tenant_id'),
"tenant_name":"GlobalScape",
"time": kwargs.get('time'),
"status_code": kwargs.get('status_code'),
"status_detail": kwargs.get('status_detail'),
"user": {
"home_directory": kwargs.get('user_home_directory'),
"uuid": kwargs.get('guid'),
"uid": kwargs.get('user_uid'),
"type": kwargs.get('user_type'), "type": kwargs.get('user_type'),
"name": kwargs.get('user_name') "name": kwargs.get('user_name')
}, },
@ -101,18 +50,16 @@ def dataTemplate(transactionType,**kwargs):
"file": { "file": {
"size": kwargs.get('file_size'), "size": kwargs.get('file_size'),
"name": kwargs.get('file_name'), "name": kwargs.get('file_name'),
"path": kwargs.get('file_path'), "path": kwargs.get('file_path')
"uid": kwargs.get('file_uid'),
}, },
"guid": f'deleted:{kwargs.get("guid")}', "guid": kwargs.get('guid'),
"node_name": kwargs.get('node_name'),
"classifications": [{ "classifications": [{
"ref_id": f"globalscape:{kwargs.get('guid')}", "ref_id": f"globalscape:{kwargs.get('guid')}",
"time": kwargs.get('time'), "time": kwargs.get('time'),
}], }],
"prd_ext_tenant_name": kwargs.get("prd_ext_tenant_name"), "prd_ext_tenant_name": "Globalscape EFT",
"prd_ext_tenant_id": kwargs.get('prd_ext_tenant_id'), "prd_ext_tenant_id": kwargs.get('tenant'),
"product_name": kwargs.get("product_name"), "product_name": "Globalscape EFT",
"session": { "session": {
"created_time": kwargs.get('time'), "created_time": kwargs.get('time'),
"uid": kwargs.get('session_uid') "uid": kwargs.get('session_uid')
@ -122,20 +69,16 @@ def dataTemplate(transactionType,**kwargs):
"ip": kwargs.get('src_endpoint_ip'), "ip": kwargs.get('src_endpoint_ip'),
"type": kwargs.get('src_endpoint_type') "type": kwargs.get('src_endpoint_type')
}, },
"tenant": kwargs.get('prd_ext_tenant_id'),
"tenant_name":"GlobalScape",
"dst_endpoint": { "dst_endpoint": {
"port": kwargs.get('dst_endpoint_port'), "port": kwargs.get('dst_endpoint_port'),
"ip": kwargs.get('dst_endpoint_ip'), "ip": kwargs.get('dst_endpoint_ip'),
"type": kwargs.get('dst_endpoint_type') "type": kwargs.get('dst_endpoint_type')
}, },
"time": kwargs.get('time'), "time": kwargs.get('time'),
"status_code": kwargs.get('status_code'),
"status_detail": kwargs.get('status_detail'),
"user": { "user": {
"home_directory": kwargs.get('user_home_directory'), "home_directory": kwargs.get('user_home_directory'),
"uuid": kwargs.get('user_session_uid'), "uuid": kwargs.get('guid'),
"uid": kwargs.get('user_uid'), "uid": kwargs.get('uid'),
"type": kwargs.get('user_type'), "type": kwargs.get('user_type'),
"name": kwargs.get('user_name') "name": kwargs.get('user_name')
}, },
@ -153,14 +96,9 @@ def dataTemplate(transactionType,**kwargs):
"type": kwargs.get('dst_endpoint_type') "type": kwargs.get('dst_endpoint_type')
}, },
"guid": kwargs.get('guid'), "guid": kwargs.get('guid'),
"node_name": kwargs.get('node_name'), "prd_ext_tenant_id": kwargs.get('tenant'),
"tenant": kwargs.get('prd_ext_tenant_id'), "product_name": "GlobalScape EFT",
"tenant_name":"GlobalScape", "prd_ext_tenant_name": "GlobalScape EFT",
"prd_ext_tenant_id": kwargs.get('prd_ext_tenant_id'),
"product_name": kwargs.get("product_name"),
"prd_ext_tenant_name": kwargs.get('prd_ext_tenant_name'),
"status_code": kwargs.get('status_code'),
"status_detail": kwargs.get('status_detail'),
"src_endpoint": { "src_endpoint": {
"port": kwargs.get('src_endpoint_port'), "port": kwargs.get('src_endpoint_port'),
"ip": kwargs.get('src_endpoint_ip'), "ip": kwargs.get('src_endpoint_ip'),
@ -169,22 +107,18 @@ def dataTemplate(transactionType,**kwargs):
"time": kwargs.get('time'), "time": kwargs.get('time'),
"user": { "user": {
"home_directory": kwargs.get('user_home_directory'), "home_directory": kwargs.get('user_home_directory'),
"uuid": kwargs.get('user_session_uid'), "uuid": kwargs.get('guid'),
"uid": kwargs.get('user_uid'), "uid": kwargs.get('uid'),
"type": kwargs.get('user_type'), "type": kwargs.get('user_type'),
"name": kwargs.get('user_name') "name": kwargs.get('user_name')
}, },
"session": {
"created_time": kwargs.get('time'),
"uid": kwargs.get('session_uid')
},
"utype": kwargs.get('utype') "utype": kwargs.get('utype')
} }
if transactionType == "file_uploaded": if transactionType == "file_uploaded":
template = upload template = uploadDownload
if transactionType == "file_downloaded": if transactionType == "file_downloaded":
template = download template = uploadDownload
if transactionType == "file_deleted": if transactionType == "file_deleted":
template = fileDeleted template = fileDeleted
if transactionType == "user_logged_on": if transactionType == "user_logged_on":

90
inexDataProcessing.py
View File

@ -1,74 +1,56 @@
def processData(data, template, **kwargs): def processData(data, template, **kwargs):
"""Translates data from sql query to the appropriate place in the respective template.
Accepts data, which is the sql query output, the template function, and finally
additional data to insert into the template. Uses other functions to further
process row data."""
processedData = [] processedData = []
transactionLoginid = [] transactionLoginid = []
for row in data: for row in data:
# print(f'Row: {row}') # print(f'Row: {row}')
# must set variables for the different templates and do logic based on that. Do not call identifyUtype many times
identifyUtypecommand = identifyUtype(row.get('Command'))
if identifyUtypecommand == "other":
continue
if row.get('Command') == None: if row.get('Command') == None:
continue continue
userType = identifyUserType(row.get('user_type'))
userHome = parseHomefolder(row.get('Actor'),row.get('VirtualFolderName'))
try: try:
processedData.append(template(identifyUtypecommand,\ processedData.append(template(identifyUtype(row.get('Command')),\
prd_ext_tenant_name=kwargs.get('prd_ext_tenant_name'),\ prd_ext_tenant_id='',\
user_uid=row.get('TransactionID'),\
status_detail=row.get('Description'),\
prd_ext_tenant_id=kwargs.get('prd_ext_tenant_id'),\
status_code=row.get('ResultID'),\ status_code=row.get('ResultID'),\
file_created_time=row.get('Time_stamp'),\
file_size=row.get('FileSize'),\ file_size=row.get('FileSize'),\
file_uid=row.get('ProtocolCommandID'),\
file_path=row.get('PhysicalFolderName'),\ file_path=row.get('PhysicalFolderName'),\
file_virtual_path=row.get('VirtualFolderName'),\
file_name=row.get('FileName'),\ file_name=row.get('FileName'),\
guid=row.get('TransactionGUID'),\ guid=row.get('TransactionGUID'),\
ref_id=row.get('ProtocolCommandID'),\
prd_instance_id=kwargs.get('prd_instance_id'),\
product_guid=kwargs.get('product_guid'),\
product_name=kwargs.get('product_name'),\ product_name=kwargs.get('product_name'),\
product_version=kwargs.get('product_version'),\
node_name=row.get('NodeName'),\ node_name=row.get('NodeName'),\
session_uid=row.get('TransactionID'),\
src_endpoint_type=row.get('Protocol'),\ src_endpoint_type=row.get('Protocol'),\
src_endpoint_port=row.get('RemotePort'),\ src_endpoint_port=row.get('RemotePort'),\
src_endpoint_ip=row.get('RemoteIP'),\ src_endpoint_ip=row.get('RemoteIP'),\
dst_endpoint_port=row.get('LocalPort'),\ dst_endpoint_port=row.get('LocalPort'),\
dst_endpoint_ip=row.get('LocalIP'),\ dst_endpoint_ip=row.get('LocalIP'),\
dst_endpoint_type=row.get('Protocol'),\ dst_endpoint_type=row.get('Protocol'),\
user_session_uid=row.get('TransactionID'),\ session_uid=row.get('TransactionID'),\
bytes=row.get('BytesTransferred'),\ bytes_out=row.get('BytesTransferred'),\
time=row.get('Time_stamp'),\
duration=row.get('TransferTime'),\ duration=row.get('TransferTime'),\
user_type=userType,\ time=row.get('Time_stamp'),\
user_type=identifyUserType(row.get('user_type')),\
user_domain=row.get('SiteName'),\
user_name=row.get('Actor'),\ user_name=row.get('Actor'),\
user_home_directory=userHome,\ user_home_directory=row.get('VirtualFolderName'),\
utype=identifyUtypecommand)) description=row.get('Description'),\
utype=identifyUtype(row.get('Command'))))
except UnboundLocalError: except UnboundLocalError:
print(f'Problem row GUID:{row.get("TransactionGUID")} ::: TransactionObject:{row.get("TransactionObject")} Command: {row.get("Command")}') print(f'Problem row GUID:{row.get("TransactionGUID")} ::: TransactionObject:{row.get("TransactionObject")} Command: {row.get("Command")}')
continue
identifyUtypetransactionObject = identifyUtype(row.get('TransactionObject'))
if identifyUtypetransactionObject == "other":
continue continue
if row.get('TransactionGUID') not in transactionLoginid: if row.get('TransactionGUID') not in transactionLoginid:
try: try:
processedData.append(template(identifyUtypetransactionObject,\ processedData.append(template(identifyUtype(row.get('TransactionObject')),\
prd_ext_tenant_id=kwargs.get('prd_ext_tenant_id'),\
prd_ext_tenant_name=kwargs.get('prd_ext_tenant_name'),\
status_detail=row.get('Description'),\
guid=row.get('TransactionGUID'),\ guid=row.get('TransactionGUID'),\
status_code=row.get('ResultID'),\
node_name=row.get('NodeName'),\
prd_instance_id=kwargs.get('prd_instance_id'),\ prd_instance_id=kwargs.get('prd_instance_id'),\
product_guid=kwargs.get('product_guid'),\
product_name=kwargs.get('product_name'),\ product_name=kwargs.get('product_name'),\
product_version=kwargs.get('product_version'),\
src_endpoint_type=row.get('Protocol'),\ src_endpoint_type=row.get('Protocol'),\
src_endpoint_port=row.get('RemotePort'),\ src_endpoint_port=row.get('RemotePort'),\
src_endpoint_ip=row.get('RemoteIP'),\ src_endpoint_ip=row.get('RemoteIP'),\
@ -76,14 +58,14 @@ def processData(data, template, **kwargs):
dst_endpoint_ip=row.get('LocalIP'),\ dst_endpoint_ip=row.get('LocalIP'),\
dst_endpoint_type=row.get('Protocol'),\ dst_endpoint_type=row.get('Protocol'),\
session_uid=row.get('TransactionID'),\ session_uid=row.get('TransactionID'),\
bytes_out=row.get('BytesTransferred'),\
transfer_time=row.get('TransferTime'),\ transfer_time=row.get('TransferTime'),\
time=row.get('Time_stamp'),\ time=row.get('Time_stamp'),\
user_session_uid=row.get('TransactionID'),\ user_type=identifyUserType(row.get('user_type')),\
user_uid=row.get('TransactionID'),\ user_domain=row.get('SiteName'),\
user_type=userType,\
user_name=row.get('Actor'),\ user_name=row.get('Actor'),\
user_home_directory=userHome,\ user_home_directory=row.get('VirtualFolderName'),\
utype=identifyUtypetransactionObject\ utype=identifyUtype(row.get('TransactionObject'))\
)) ))
transactionLoginid.append(row.get('TransactionGUID')) transactionLoginid.append(row.get('TransactionGUID'))
except UnboundLocalError: except UnboundLocalError:
@ -93,7 +75,6 @@ def processData(data, template, **kwargs):
return processedData return processedData
def identifyUserType(obj): def identifyUserType(obj):
"""Check string if it has Admin-> return Administrator else return User."""
if obj: if obj:
if "Admin" in obj: if "Admin" in obj:
return "Administrator" return "Administrator"
@ -101,24 +82,7 @@ def identifyUserType(obj):
return "User" return "User"
else: else:
return None return None
def parseHomefolder(user, virtualfolder):
"""Extract users home folder using the username. Will not work on edge cases
such as when a users home folder does not have the user name. When that occurs
it is impossible to know based on the arm data what the home folder is.
This function is an assumption so it may return the incorrect home folder.
This function finds the user name and takes the path from the left of the folder
as the home folder. There are cases where this may not be accurate."""
if user:
userSplit = f'/{user}/'
if virtualfolder:
if userSplit in virtualfolder:
home = virtualfolder.split(userSplit)[0] + userSplit
return home if home else None
def identifyUtype(obj): def identifyUtype(obj):
"""Process Type of transaction based on string that passed in.
Return transaction type."""
user_logged_on = ['AUTH'] user_logged_on = ['AUTH']
file_deleted = ["dele"] file_deleted = ["dele"]
file_uploaded = ["created"] file_uploaded = ["created"]
@ -126,11 +90,11 @@ def identifyUtype(obj):
if obj in user_logged_on: if obj in user_logged_on:
return "user_logged_on" return "user_logged_on"
elif obj in file_deleted: if obj in file_deleted:
return "file_deleted" return "file_deleted"
elif obj in file_uploaded: if obj in file_uploaded:
return "file_uploaded" return "file_uploaded"
elif obj in file_downloaded: if obj in file_downloaded:
return "file_downloaded" return "file_downloaded"
else: else:
return "other" return "other"

2
inexEncoder.py
View File

@ -3,8 +3,6 @@ import decimal
import datetime import datetime
class Encoder(json.JSONEncoder): class Encoder(json.JSONEncoder):
"""Encoder uses json.JSONEncoder and checks for instances of decimal and datetime.
Changes decimal.Decimal to int and datetime.datetime to unix timestamp with miliseconds."""
def default(self, o): def default(self, o):
if isinstance(o, decimal.Decimal): if isinstance(o, decimal.Decimal):
return int(o) return int(o)

8
inexSqlquery.py
View File

@ -1,11 +1,11 @@
class sqlQuerymodel: class sqlQuerymodel:
def queryData(overRideflag, configQuery, daysTopull): def queryData(overRideflag, configQuery):
"""Embedded query data. Data is slightly modified to change the amount of days to pull.""" """Embedded query data"""
q ="""DECLARE @stopTime DATETIME2 q ="""DECLARE @stopTime DATETIME2
SET @stopTime=DATEADD(DAY, -30, GETDATE()) SET @stopTime=DATEADD(DAY, -30, GETDATE())
SELECT p.ProtocolCommandID, t.Time_stamp, p.RemoteIP, p.RemotePort, p.LocalIP, p.LocalPort, p.Protocol, p.SiteName, p.Command, p.FileName, p.PhysicalFolderName, p.VirtualFolderName, p.FileSize, p.TransferTime, p.BytesTransferred, p.Description, p.ResultID, t.TransactionID, p.Actor, t.TransactionObject, t.NodeName, t.TransactionGUID, a.Protocol user_type SELECT p.ProtocolCommandID, t.Time_stamp, p.RemoteIP, p.RemotePort, p.LocalIP, p.LocalPort, p.Protocol, p.SiteName, p.Command, p.FileName, p.VirtualFolderName, p.FileSize, p.TransferTime, p.BytesTransferred, p.Description, p.ResultID, t.TransactionID, p.Actor, t.TransactionObject, t.NodeName, t.TransactionGUID, a.Protocol user_type
FROM tbl_Transactions t FROM tbl_Transactions t
Full JOIN tbl_ProtocolCommands p ON(t.TransactionID=p.TransactionID) Full JOIN tbl_ProtocolCommands p ON(t.TransactionID=p.TransactionID)
Full join tbl_Authentications a ON(t.TransactionID=a.TransactionID) Full join tbl_Authentications a ON(t.TransactionID=a.TransactionID)
WHERE p.Time_stamp>@stopTime AND p.Command IS NOT NULL""".replace("30", str(daysTopull)) WHERE p.Time_stamp>@stopTime AND p.Command IS NOT NULL"""
return configQuery if overRideflag else q return configQuery if overRideflag else q

31
test.py
View File

@ -15,34 +15,7 @@ def builddict(keys,*args,**kwargs):
dict[key] = kwargs.get(key) dict[key] = kwargs.get(key)
print(dict) print(dict)
testfolder = '/Usr/a/asdf/asf'
def identifyUtype(obj):
"""Process Type of transaction based on string that passed in.
Return transaction type."""
user_logged_on = ['AUTH']
file_deleted = ["dele"]
file_uploaded = ["created"]
file_downloaded = ["sent"]
if obj in user_logged_on:
return "user_logged_on"
if obj in file_deleted:
return "file_deleted"
if obj in file_uploaded:
return "file_uploaded"
if obj in file_downloaded:
return "file_downloaded"
else:
return "other"
testfolder = '/Usr/a/asdf/asf/asdfas/asdfasdf/'
user = 'a' user = 'a'
def parsehomefolder(user, virtualfolder): print(testfolder.split(f"/{user}/"))
userSplit = f'/{user}/'
home = virtualfolder.split(userSplit)[0] + userSplit
print(home)
return home
a = parsehomefolder(user, testfolder)