Salesforce, Python, SQL, & other ways to put your data where you need it

Need event music? 🎸

Live and recorded jazz, pop, and meditative music for your virtual conference / Zoom wedding / yoga class / private party with quality sound and a smooth technical experience

Help: Baby pen test?

24 Mar 2022 🔖 security devops
💬 EN

Table of Contents

Almost 2 years after I said I’d like to get involved in a project at work that involves taking a tool comprised of a lot of databases and web servers and putting them into “the cloud,” I’m still not really involved at a command line. But I’ve gotten to be a fly on the wall and a translator between teams with different realms of expertise, and I’ve been invited to help our internal security experts ensure that logging is meaningful, to facilitate anomaly detection. Can you help me think of more things to test? I’m not an expert in all the “doors that should be locked” when it comes to Linux servers and database servers.

Basic logging questions

Here are the basic questions I think should be asked of every single one of the “Can we see ourselves do this?” activities I’ve thought of:

  1. What does the attempted activity look like in centralized logs that are regularly monitored by security specialists?
  2. If it shows up in those logs, is it labeled well enough to be meaningful? (Like adding ethyl mercaptan to natural gas & propane!)
  3. If it doesn’t show up in those logs, what would have to be done to make it show up (in a way that can be labeled meaningfully)?
  4. If there’s no way to make the attempted activity show up in centralized logs, or if the attempted activity happens so frequently that it would overwhelm day-to-day centralized logs, what can be done to make sure that the information is either regularly delivered to centralized security teams through another channel or that it’s available to be delivered to them (both as a “historical report” and as a “real-time feed”) with about 5 minutes of effort during an emergency?
    • (Possibly into the centralized logs during an emergency, just by “flipping on a switch” that was turned off for normal operations – and knowing that the groundwork was laid that when the activities start flowing to the logs, it has been planned and tested ahead of time that they will arrive meaningfully labeled.)

Definition of meaningful

Here are some things that make me feel like log events are “labeled meaningfully”:

  1. We know the IP address of all computers involved in the activity (not just “from” but also “to” and maybe even “via”).
  2. If one of our cloud servers is involved, we know the domain name people think of it as (e.g. “mobileapp.ourcompany.com“), and perhaps also any intermediate domain names that such traffic might pass through behind the scenes.
  3. If one of our cloud servers is involved, we see a quick-reference “nickname” we made up for it (e.g. “primary_database_server” or perhaps its resource name as provisioned in our cloud).
  4. If one of our corporate-issued desktops/laptops is involved, we know the hardware-management asset ID of the computer that performed the activity.
  5. If a recognizable-username-based connection is involved (like logging into our VPN, logging into username-based on-site wifi, or logging into the operating system of one of our corporate-issued desktops/laptops), we know what username(s).
  6. If a username for the operating system of a cloud server is involved, we know what username.
  7. If a username for a database hosted in the cloud is involved, we know what username.
  8. If a username for an unprotected system is involved as a destination (e.g. trying to SSH from one of our infrastructure servers to a “foreign” server, or trying to query a database hosted on a “foreign” server from within one of our infrastructure databases), we know what username we were trying to “reach out” to the foreign system as.
  9. We see a quick-reference “nickname” for the protocol and/or the “category” of activity (e.g. “SSH”, “database connection”, “database sensitive activity”, “file operation”, etc.) that’s easy to use for searching, sorting, filtering, etc.
  10. We see a quick-reference “nickname” for this entire infrastructure as a whole (e.g. “super_duper_system”) that’s easy to use for searching, sorting, filtering, etc.

Activity ideas

SSH

From a corporate desktop that sits physically on-site, to a Linux server in the cloud:

  • SSH in, using both valid and invalid user credentials
    • When using valid credentials, poke around a while before disconnecting, and observe the “poking around” and the “disconnect time” against the basic logging questions, not just the moment of login.
  • Repeat for all sorts of servers, to help test for “is the labeling meaningful?” (Can we easily tell which server it was from centralized logs?)

Then do the same “Hello? Can we see ourselves do this?” SSH attempt with the following alternate sources and destinations:

  1. From a corporate laptop at someone’s home w/o VPN, to a Linux server in the cloud.
  2. From a corporate laptop at someone’s home with VPN, to a Linux server in the cloud.
  3. From a personal computer at someone’s home w/o VPN, to a Linux server in the cloud.
  4. From a personal computer at someone’s home with VPN, to a Linux server in the cloud.
  5. From a personal laptop physically on-site plugged into an ethernet jack, to a Linux server in the cloud.
  6. From a personal laptop physically on-site using username-and-password-authenticated WiFi, to a Linux server in the cloud.
  7. From a personal laptop physically on-site using guest WiFi, to a Linux server in the cloud.
  8. From a Linux server in the cloud to a Linux server in the cloud (both servers from within the same project, but may or may not be expected to talk to each other under normal operations).
  9. From a Linux server in the cloud to a Linux server in the cloud with a corporate IP address but from an unrelated project.
  10. From a Linux server in the cloud to a corporate desktop that sits physically on-site.
  11. From a Linux server in the cloud to a personal laptop physically on-site plugged into an ethernet jack.
  12. From a Linux server in the cloud to a personal laptop physically on-site using username-and-password-authenticated WiFi.
  13. From a Linux server in the cloud to a personal laptop physically on-site using guest WiFi.
  14. From a Linux server in the cloud to a Linux server that looks like it’s randomly elsewhere on the internet (e.g. at someone’s home, or rented from the cloud but not given a corporate IP address, etc.)

Other network tests

  1. Same ideas as “SSH”, but ping.
  2. Same ideas as “SSH”, but curl or Postman (goes over HTTP).
    • Both to endpoints that servers may well intend to be hosting and to endpoints that they’re not trying to host.
    • Some known endpoints may be meant for web browsers w/ humans; others might be meant for server-to-server communication like POST requests.
  3. Same ideas as “SSH”, but with GUI-based and CLI-based tools (as appropriate to the source machine) for logging into database management systems hosted on these Linux servers, rather than trying to connect to the Linux OS itself.
    • Don’t forget that in addition to the “from-to” flows listed under “SSH,” there is also a question of from a Linux server in the cloud’s own operating system environment to the database management system it hosts. That wouldn’t traverse a network connection, so the Linux OS or DBMS in question is going to have to “tattle on itself” to make this activity show up in centralized logs.
  4. Same ideas as “SSH”, but with a web browser … legitimately visit a web site hosted by a known web server.
    • Try both correctly and incorrectly authenticating as both “human” and “departmental” users, as applicable.
    • Try all kinds of web servers – ones aimed at the general public, ones aimed only at internal power users as configuration GUIs, etc.
    • When using valid credentials, poke around a while before disconnecting, and observe the “poking around” and the “disconnect time” against the basic logging questions, not just the moment of login. Make sure to do some “poking around” that should have side effects on other servers, like looking up data from a database. How good does the inter-server logging look for that activity?
  5. Same ideas as “SSH”, but with specialized desktop software meant for interacting with particular specialized servers.

Database tests

Network

Log into a database as a valid user with appropriate privileges. Execute a query that fetches data from a database hosted on another server (e.g. in Oracle this would be via DBLink).

  • If “known links” can be queried, as they can with Oracle, write a script that loops over all “known links” and tries them. Don’t forget this script can also be a useful source of log creation, if written to be one.

Local

Log into a database as a valid user with appropriate superpowered privileges. Do a bunch of sensitive operations such as:

  • Create a new database user.
  • Grant access to query an existing table to an existing database user.
  • Delegate superpowers to an existing database user who doesn’t already have them.
  • Take important administrative and disaster-recovery permissions away from an existing database user.

These activities wouldn’t traverse a network connection, so the database management system in question is going to have to “tattle on itself” to make this activity show up in centralized logs.

Linux server tests

  1. Create a file.
  2. Rename a file.
  3. Edit a file.
  4. Delete a file.
  5. Change the “group” of a file.
  6. Change the “owner” of a file.
  7. Change the “permissions” of a file.
  8. Perform a single operation as another user (su / sudo, I think?).
  9. Shift to acting as another user for all commands that follow (su / sudo, I think?) until exited. Poke around for a while before exiting privilege escalation, both with innocuous commands like echo and with potentially sensitive commands like the file tests above.

These activities wouldn’t traverse a network connection, so the Linux OS in question is going to have to “tattle on itself” to make this activity show up in centralized logs.

Question for readers

What’d I miss?

--- ---